ITSG-33 security controls mapping for TEF V4 - for org policies

20240524: I  will update the subtasks later today
work items

us as a reference - the 12 guardrails minimum 30-day security for profile 3 (details in a lower comment

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#guardrails-evidence-package

https://github.com/canada-ca/cloud-guardrails/blob/master/EN/00_Applicable-Scope.md#table2-guardrail-identifiers-service-models-and-cloud-usage-profiles

https://github.com/canada-ca/cloud-guardrails/blob/master/EN/13_Plan-for-Continuity.md

map existing TEF controls to KCC - as a discovery exercise only

get security team member to drive deliverables

work through existing 6 policies - https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/1-org/envs/shared/org_policy.tf#L118C1-L129C2

determine org policies missing - optionally add them to org_policy

worry about dev runtime folder overrides later - for now turn off policies during dev

yhe gke policies for the gateway controller are optional https://github.com/terraform-google-modules/terraform-example-foundation/tree/master/policy-library/policies/constraints

reference 66 contols in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/wiki/Security-Controls - break out only org policy ones

Start with https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/wiki/Security-Controls
specifically the org policies (doc work)

lz v2 - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone/org/org-policies
tef - https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/1-org/envs/shared/org_policy.tf#L64

work items

correlate the TEF org_policy with control mapping in KCC and the NIST-800-53
SCC-p has controls - output
http://go/91-controls see mappings
check

example locationRestriction

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#6260sc-7boundary-protection
SC-7
see https://cloud.google.com/resource-manager/docs/organization-policy/creating-managing-custom-constraints
reference 3 part regionalization in upcoming upstream sync https://github.com/terraform-google-modules/terraform-example-foundation/pull/1181/files

Example incident response documents

https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident/?e=48754805

Discussion notes to incorporate above
We should be ok with what is already in the TEF, what is inherent in GCP already (encryption, IAM role restrictions...), what will be inherent in the client's AD identity provider (we supply what workspace does for MFA/2FA for example - the client correlates this via their AD identities as an analog IdP).
For non-federal full ITSG-33 (PBMM/FedRAMP) will not be required but is useful.  A request for the client security team to get involved early on is useful.  but we can prepare a minimum default set of compliance documentation - starting with a wiki/readme on our end.
I am not that concerned about the org policies, the 4 in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/1-org/envs/shared/org_policy.tf are a start.
I expect we will not need custom policies - as the out of the box org policies (including the 4-6 set by GCP by default should be ok, IP related, location restriction, service account lockdown, shielded VMs...).  I don't think we will need to get into the rego that is used by KCC.  KCC is just a guide of where we could go.

 Bottom line we need at least the 30 day guardrails level of security - at profile 3.
 About interconnects - this is profile 5/6 and is workload related - it can be in phase 2

 We can start with 3 documents
 1) The guardrails evidence section of the (federal) document at

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#guardrails-evidence-package
If you check the details of the GR's - they are around monitoring, alerts, VPC firewall rules/segmentation, accounts, encryption at rest/transit, the waf/armor, and the problematic marketplace restriction (2 more org policies).
The NGFW's handle a lot of network, security for us - we just reference some evidence of traffic capture, enable/disable for east-west and north-south canary loads.
 2) The guardrails themselves (the older version has a profile page) - we make sure we have at least 1 piece of evidence for required GR 1-12.
 In the end the TEF is already mentioned as FedRAMP compliant - we know this depends on what is enabled and it is up to us to gather evidence.

https://github.com/canada-ca/cloud-guardrails/blob/master/EN/00_Applicable-Scope.md#table2-guardrail-identifiers-service-models-and-cloud-usage-profiles
For example GR13 is new - https://github.com/canada-ca/cloud-guardrails/blob/master/EN/13_Plan-for-Continuity.md
The evidence would be a procedure for a 2nd super admin account (the break glass) - where the MFA and password are kept locked/separate
 If any rule was a base default it would be GR5 - location https://github.com/canada-ca/cloud-guardrails/blob/master/EN/05_Data-Location.md#mandatory-requirements
 In reality we can usually get a waiver per workload - at the non-federal level - for services like NCC, or some the AI services - that do not reside in CA yet (but are in the plan) and currently need US locations overrides.  Here we mention intent, have the policy in place in the yaml - but not necessarily enable it on our test deployment.

for example, guardrails navigation for screen caps for the location restriction

deploy/modified tf code in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/1-org/envs/shared/org_policy.tf#L100C41-L101C84

navigate as reference to

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#guardrails-subset

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#05-data-location-1

check list

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#05-data-location-1

navigate to GR05

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#6260sc-7boundary-protection

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/z_2024_v020_pre_tef_v4/docs/google-cloud-security-controls.md#iam---organization-policies---resource-location-restriction

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/2-environments/modules/env_baseline/monitoring.tf

runtime example of loggin https://console.cloud.google.com/logs/query;cursorTimestamp=2024-05-27T14:16:27.851800721Z;duration=PT1H?referrer=search&hl=en&orgonly=true&project=prj-d-monitoring-hwfg&supportedpurview=project

see #452


Include co2 and cost savings on baseload insights notice

@ldodds confirmed with Philip so I think this one's ready for review now.


Consignment validation fails with single Stock/Inventory

According to my comment here, I found a bug when I try to validate the consignment file of the second transfer.
Consignment is NOT valid
Validation failures:
- seal b21d151cf898f451ac24edc5e0881cc1c9fa70bf869b3d6505d1b4b76289c718/0/1 present in the history is confidential and can't be validated.
Validation warnings:
- duplicated terminal seal AYaFxoGfjxTWBzhsEBwsiL7h9JAsnfWQfsgzqgjDuTna in operation 7130d1e33555df49b31b0cf48ce1c1e9fc2feb7b1b41a713dee5cdb83d927eab.

Important: This error occurs only when we create an invoice and a transfer using a single stock/inventory. When we use different persistence stocks, everything works!

Can you please send be the consignment - and also the stock.dat file? That would really help in debugging

I will work on this issue independently from the merge of https://github.com/RGB-WG/rgb-wallet/issues/36#issuecomment-1531104393 since it has lesser criticality


Bug fix userinfo component decoded

try to resolve #28
Only the following methods are affected:

UserInfo::createFromUri;
UserInfo::createFromAuthority;
UserInfo::withContent;

The expected data is supposed to be encoded and decoding will be done as to avoid double encoding.

@Gounlaf the following PR should resolve your issue. Can you test it on your code to see if this is what you expect.
thanks in advance.

@Gounlaf the following PR should resolve your issue. Can you test it on your code to see if this is what you expect.
thanks in advance.

@nyamsprod it's working as expected :)


Implement healthchecks.io in Kubernetes

https://github.com/heywoodlh/flakes/commit/ae4ad97c7b8a8fc27ae3bd7a9f20cdc7c24659ed


Allow multiple labels to be passed in the update comand

https://github.com/github/hub/issues/2838

Sorry, but your workaround as discovered here is the proper fix for the issue: https://github.com/github/hub/issues/2838#issuecomment-934290814
We don't want to change the flag parsing code to accommodate spaces. All values with spaces need to be enclosed in quotes (or escaped by other means), always.


Linting, CI and Makefile

Added compilation directives to Martin's original work on CI. Moved dependicies to "third_party"

(but please finish your discussion with @local-minimum before merging!)


Minor touch ups

@justinmayhew Due to how long this PR has been opened for, I could not squash properly, thus I implemented the changes you made here and released: https://github.com/johnnyji/proper_case/pull/29
Thank you for your contribution!

Thanks @johnnyji!


[BUG] Agent v7.40.0 fails to get hostname from Kubernetes causing a crash loop

After Keel upgraded from 7.39.2 today at 12:11PM ET - we now have our pod failing to start because it cannot retrieve a hostname.
Logs from the agent pod attached.
Agent Environment
datadog-agent-failing-to-get-hostname.txt
Describe what happened:
Upgrading to 7.40 via Keel - running image
docker-pullable://public.ecr.aws/datadog/agent@sha256:05305e303d5056aa4c4919b77019926149dfc497031e062ba965c46734768cde
Describe what you expected:
Hostname to be correctly identified.
Steps to reproduce the issue:
Run the above docker image under kubernetes. Specifically EKS for us.
Additional environment details (Operating System, Cloud provider, etc):
AWS EKS v1.21.14
Using public docker image

Seeing similar behavior, our CI is reporting:

Error: Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file

And then crashing with:

Error: unable to read authentication token file: open /etc/datadog-agent/auth_token: no such file or directory

Seems like the release notes touch on this; https://github.com/DataDog/datadog-agent/compare/7.39.2...7.40.0#diff-cf2b4696563d13f55549abf65ac5a23d2e280777f03753acdce2dd52d38498fc

This also happens when running as a container in an EC2-ECS task. (Fargate-ECS seems fine).

same error here, running the deployment below:
apiVersion: datadoghq.com/v1alpha1 kind: DatadogAgent metadata:   name: datadog spec:   credentials:     apiSecret:       secretName: datadog-secret       keyName: api-key     appSecret:       secretName: datadog-secret       keyName: app-key   agent:     config:       collectEvents: true     image:       name: "gcr.io/datadoghq/agent:latest"   clusterAgent:     config:       collectEvents: true     image:       name: "gcr.io/datadoghq/cluster-agent:latest"
2022-11-03 05:44:56 UTC | CORE | INFO | (pkg/util/log/log.go:588 in func1) | runtime: final GOMAXPROCS value is: 2 2022-11-03 05:44:56 UTC | CORE | WARN | (pkg/util/log/log.go:616 in func1) | Agent configuration relax permissions constraint on the secret backend cmd, Group can read and exec 2022-11-03 05:44:56 UTC | CORE | INFO | (pkg/util/log/log.go:588 in func1) | Features detected from environment: kubernetes,cri,containerd 2022-11-03 05:44:56 UTC | CORE | INFO | (cmd/agent/app/run.go:257 in StartAgent) | Starting Datadog Agent v7.40.0 2022-11-03 05:44:57 UTC | CORE | ERROR | (cmd/agent/app/run.go:313 in StartAgent) | Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file 2022-11-03 05:44:57 UTC | CORE | INFO | (pkg/logs/logs.go:149 in Stop) | Stopping logs-agent 2022-11-03 05:44:57 UTC | CORE | INFO | (pkg/logs/logs.go:158 in Stop) | logs-agent stopped 2022-11-03 05:44:57 UTC | CORE | INFO | (cmd/agent/app/run.go:535 in StopAgent) | See ya! Error: Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file

Hello,
The change is indeed intended as explained in the CHANGELOG.
Having no proper hostname means that container name/pod name is used as hostname, which can lead to many features not working properly (poor visibility in Infrastructure list, inability to link with cloud integrations, etc.).
On Kubernetes, it usually means the Agent cannot access at least one of:

Cloud provider metadata endpoint
Kubelet API
Docker runtime (if used)
Which means important features (tagging, autodiscovery, cloud integrations) won't work.

Check recommended Kubernetes setup for each major distribution at:
https://docs.datadoghq.com/containers/kubernetes/distributions
Kubernetes
Solution 1: Make sure the Agent is able to access the Kubelet API. This is usually handled automatically by our Helm chart/Operator.
@JamieSinn It seems like you're not using Helm/Operator, check that the Role/ClusterRole assigned to the Agent has proper RBAC to access Kubelet API:
https://github.com/DataDog/helm-charts/blob/125a47e7f9ba9146499c0f5e87d42e06c1e19110/charts/datadog/templates/rbac.yaml#L70-L78
Solution 2: @andreslon On AKS (see distribution link above), you need to set (Operator):
datadog:
  kubelet:
    tlsVerify: false # Required as of Agent 7.35. See Notes.

Because Kubelet certificates on AKS are not properly signed (SAN missing).
Solution 3: Should only be necessary if you explicitly don't want the Agent to connect to Kubelet API (unlikely).
In this case you can use the downward API to set DD_HOSTNAME:
      env:
        - name: DD_HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

ECS EC2
@cmwilhelm It means the Agent cannot access metadata API nor Docker socket.
You'd need to check why metadata API is not accessible (maybe you explicitly forbid access).
But the easiest is to make sure the Docker socket is mounted as explained in our reference manifest: https://docs.datadoghq.com/resources/json/datadog-agent-ecs.json
Docker VM/CI/etc.
The Docker socket should be mounted as explained in our documentation:
-v /var/run/docker.sock:/var/run/docker.sock:ro

When running in a CI environment (e.g. Agent is not ephemeral), where the hostname is not really important, simply set DD_HOSTNAME to a value:
-e DD_HOSTNAME=$(hostname)
or
- e DD_HOSTNAME=<my_hardcoded_hostname>

Thanks for that info; we are using helm - but it seems like the version updates to the actual docker image are being overwritten by Keel.
We were using an older version of the chart; which didn't have the env var set for the hostname.
I've marked the helm chart as ignore for keel; hopefully this fixes it in the future!
Thanks a lot!

We can only blame ourselves for deploying datadog-agent from the latest tag and having the datadog sidecar container marked as "essential" (ECS default). Still, releasing a breaking change as a minor version bump with just a small note at the end of the release notes is not exactly good practice IMO. :/

Thanks for the detailed response @vboulineau. However, please consider saving these kinds of breaking changes for major version updates only in the future. Even if the agent was running in a slightly degraded fashion in <7.40.0, it doesn't make sense to deliberately break it altogether without warning -- such as a semver boundary at v8.

Happened the same to us, we solved it by pinning the previous version in the installation script. One just needs to adde the following env variable on where you execute the installation of the DD Agent, this will then use the previous minor version, which in this case is 7.39.2
DD_AGENT_MAJOR_VERSION=7 \
DD_AGENT_MINOR_VERSION=39

I agree with @lifeofguenter. And even if it was better communicated, I would still expect a fallback, not crashing hard.

As per everyone else's sentiments; we run this in numerous places, from CICD to private orchestration platforms where this change broke our deployments on a "minor" release cadence.
Aside from the lost time in debugging this, only to eventually decide it must be a DD bug and come here; Now what we end up with is a massive user base of people pinning to v7.39 for eternity, which will surely only cause headaches for datadog in the long run, who will be left having to support users later down the line who never upgraded as the result of this change?

Same issue on Azure Container Apps.
Workaround using: datadog/agent:7.39.2

Here is how to implement the DD_HOSTNAME workaround using helm:
  # datadog.env -- Set environment variables for all Agents
  ## The Datadog Agent supports many environment variables.
  ## ref: https://docs.datadoghq.com/agent/docker/?tab=standard#environment-variables
  env:
    - name: DD_HOSTNAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
  #   - name: <ENV_VAR_NAME>
  #     value: <ENV_VAR_VALUE>

This results in the datadog agent correctly determining the hostname:
luke@ubu:~$ kubectl logs -n datadog-aue-dev            datadog-monitoring-linux-9sfpf -c security-agent
2022-11-18 00:57:52 UTC | SECURITY | WARN | (pkg/util/log/log.go:616 in func1) | Agent configuration relax permissions constraint on the secret backend cmd, Group can read and exec
2022-11-18 00:57:52 UTC | SECURITY | INFO | (pkg/util/log/log.go:588 in func1) | Features detected from environment: containerd,kubernetes,cri
2022-11-18 00:57:52 UTC | SECURITY | INFO | (app/app.go:265 in RunAgent) | Hostname is: aks-auedevscoa-30506518-vmss00006n

This is entirely undocumented and I had already decided to abandon upgrading to 7.40 before checking this out as one last thing.
It is extremely frustrating that the issue is not being acknowledged here and apparently no sufficient testing has taken place of this with AKS and helm as there is no useful guidance here and I have also not been able to get any useful guidance from a support ticket.
This is not what I expect from a supported paid for product.

Same issue here using helm + EKS :/ The TLS configuration didn't solve the problem for me.
I had to manually rollback the cluster agent version:
kubectl set image deployments/prod-datadog-cluster-agent cluster-agent=gcr.io/datadoghq/cluster-agent:7.39.2

    host:
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    hostCAPath: /etc/kubernetes/certs/kubeletserver.crt
    tlsVerify: false

this Helm config works on AKS, but it doesn't work with security-agent.
After 7.40 security-agent will fail with Error: Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file

EKS (AWS) user here. I've been fighting for weeks with Datadog support on an EKS 1.24 migration. Seems like cluster name / nodename autodetection stuff or whatever doesn't work very well atm in EKS 1.24 (K8S 1.24 has removed dockershim, which Datadog agent relies on). So I end up here because, you know, support ask you to run the last version everytime. My advices for my fellow K8S admin here, don't try to upgrade to this Datadog version + to K8S 1.24 at once, the problem just add up it's a PITA to handle really.
Anyway. On my "old" EKS 1.23 I've tried to upgrade to agent 7.40+ (from chart 3.1.11 to 3.5.2) as well and it doesn't work.
I think I've found why however. So a bit of context, my EKS cluster is called staging_v2
a) Datadog is constructing the ${node-name}-${cluster-name} hostAlias, as explained here https://docs.datadoghq.com/agent/guide/kubernetes-cluster-name-detection/
b) there are functions to replace "_" by "-" in the cluster name in the code
https://github.com/DataDog/datadog-agent/blob/7.40.1/pkg/util/kubernetes/clustername/clustername.go#L77
https://github.com/DataDog/datadog-agent/blob/7.40.1/pkg/util/kubelet/hostname.go#L48
c) but this naming pattern isn't consistent, it's only applied in datadog-agent it seems
c1) the datadog-cluster-agent doesn't apply this "_" to "-" replacement. Resulting in inconsistency between hosts appearing in Infrastructure list panel in chart 3.1.11 that must probably count as different hosts on the bill I won't be surprised. ⚠ and ALSO the root cause of the issue of what datadog-cluster-agent is not starting in 3.5.2

c2) the datadog-cluster-check override this logic at all by setting up the variable DD_HOSTNAME. See the link below
https://github.com/DataDog/helm-charts/blob/datadog-3.5.2/charts/datadog/templates/agent-clusterchecks-deployment.yaml#L180
resulting in the error in the logs sending you there https://github.com/DataDog/datadog-agent/blob/main/docs/agent/hostname_force_config_as_canonical.md (and btw because of the host naming pattern of EC2, the @lerlacher-haylix workaround doesn't work on EKS)

I hope my post will get software engineers attention. I'm fed up of the support "gatekeeping" on this really 🙏

Hi @damienleger can you go ahead and open a Support Ticket for this so we can look at it further? Thanks.

Hi @damienleger can you go ahead and open a Support Ticket for this so we can look at it further? Thanks.

hello @aliciascott, I had one (https://help.datadoghq.com/hc/en-us/requests/1004544) but closed in after opt in for override manually clusterName for staging-v2 (and production-v2 for my other cluster) in values.yaml. This results in various mixes of "-" and "_" in tags unlike before (see below) but at least the helm chart now working correctly. It seems impossible to have support tickets escalated to expert team; I gave up tbh and opted for workaround instead.

Hi @damienleger,
About the discrepancy issue that you describe for hostname  and kube_cluster_name between agent and cluster-agent in the comment https://github.com/DataDog/datadog-agent/issues/14152#issuecomment-1354538056.
In your case the hostname is properly retrieved but the same "normalization" logic is not applied everywhere.
So, I think it is not related to the initial bug tracked by this issue. However I do agree that what you are describing looks like a bug. I looked at the support ticket that you have mentioned (https://help.datadoghq.com/hc/en-us/requests/1004544), and I think I have some idea about where it comes from. If you like you can open a dedicated github issue with https://github.com/DataDog/datadog-agent/issues/14152#issuecomment-1354538056 in description, so that we can track separately your issue from this one. And please ping me in the new issue. (In any case I will add also a comment to the support ticket)

@clamoriniere hello, actually I've managed to do my EKS 1.24 migration. I've created an EKS without "" in the name, but with "-" instead. And for some reason since automatic cluster name detection didn't work anymore I had to hardcode clustername in the helm values.yml. But anyway, now cluster is migrated and I don't have discrepancy issue anymore since there are no more ""! More a workaround than a fix, but now I'm good with this issue. Up to you to create issue and fix if you want. I spent too many time on this, now I'm moving on :)

Hello,
We've published a page dedicated to the configuration issues that could lead to have this error with Agent >=7.40.0.
It lists all the common cases and options available to fix the Agent setup/configuration.
It's available at: https://docs.datadoghq.com/agent/troubleshooting/hostname_containers
It also documents a new option, available since Agent >=7.42.0, because there are few cases where hostname cannot be retrieved AND hostname returns a correct value (for instance, LXC containers).
In these scenarios, you'll need to explicitly set the following option:
DD_HOSTNAME_TRUST_UTS_NAMESPACE (or hostname_trust_uts_namespace in datadog.yaml)

(Technically speaking, it will instruct the Agent to trust hostname retrieved from non-root UTS namespace, which will mimic the behaviour pre-7.40).

@vboulineau, can you please specify how to set DD_HOSTNAME_TRUST_UTS_NAMESPACE if deploying Datadog in Kubernetes via the datadog-operator?

You can set env var DD_HOSTNAME_FILE to "/etc/hostname"

I got it resolved by adding the DD_HOSTNAME in the env part of datadog section.
...
datadog:
  ...
  env:
    - name: DD_HOSTNAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
  ...

I am doing Helm setup on AKS cluster.
This might help someone so keeping reference for it.
Ref: https://docs.datadoghq.com/agent/troubleshooting/hostname_containers/?tab=helm#pagetitle


feat(DB): Add methods to get_dbc_attribute

Add get_dbc_attribute method for cluster,
frame, ECU, and signal objects.
Add a test and DBC containing attributes.

[X] This contribution adheres to CONTRIBUTING.md.
[X] New tests have been created for any new features or regression tests for bugfixes.
[X] tox successfully runs, including unit tests and style checks (see CONTRIBUTING.md).

Coverage decreased (-0.08%) to 67.738% when pulling 7926704a2d07e1c251364913feeb37134a415530 on d-bohls:addGetDBCAttribute into 779b77c9df209c79ea80278be47e0779bb95f172 on ni:master.

I now see that integration tests don't affect code coverage. I'm going to add a unit test.

We track integration coverage separately from unit testing coverage. The combination of the two is the important thing.

From: Damon Bohls<EMAIL_ADDRESS>Sent: Monday, February 5, 2018 1:10:21 PM
To: ni/nixnet-python
Cc: Subscribed
Subject: Re: [ni/nixnet-python] feat(DB): Add methods to get_dbc_attribute (#224)
I now see that integration tests don't affect code coverage. I'm going to add a unit test.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ni_nixnet-2Dpython_pull_224-23issuecomment-2D363189197&d=DwMCaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=MEqk6FXoW10kfjnaOLm6bA&m=61J3Wkti0rjudNkMW4rhNLu4kE3sMAR8d-_XAPrCaj4&s=ECj9jatzpdYp9GMT13x4fU2fpTQiV-7Ih2jW4w0lO34&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AADuIe6Ofmp-5FS20VC6kpB6XsGZD-5F-5Fpoxks5tR1IdgaJpZM4R4-2DNy&d=DwMCaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=MEqk6FXoW10kfjnaOLm6bA&m=61J3Wkti0rjudNkMW4rhNLu4kE3sMAR8d-_XAPrCaj4&s=gVV-i4tMtCfYEDdCRkr0rtpuoABsOrBcNNEuQ9MCmiM&e=.


Update partner-biocatch.md

Replacing char with proper ASCII space character

@mistermik : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

#sign-off


Output Folder is empty

Python command
python covfefe.py -i input_data -o output_folder -p lex

and in my input folder there is only one file with following lines
Hi How are you
I'm good

and in my output folder it creates lexicosyntactics folder but nothing is there inside that folder.
Let me know if I'm missing anything

This looks right to me. Are there any errors in the output when you run that command?

Nope

The only logs are output to stdout. Can you paste the output when you run the command?

Sorry for the delay. There is nothing

Does your input file have the extension .txt?

+1 for checking the file extension. The fact that there were no warnings or progress bar makes me think that the file_finder didn't find any input files.

@luckytoilet @daniyall  Thanks my file didn't have txt extension.
But now fixing that leads to me a new issue:
anandhpe_ualberta_ca@cpu8ram30:~/installs/GitHub/COVFEFE$ python3 covfefe.py -i input_data -o output_folder -p lex
  0%|                                                                                | 0/1 [00:00<?, ?it/s]INFO:utils.logger:lexicosyntactic: Starting input_data/file1.txt
ERROR:utils.logger:lexicosyntactic: Failed with error

===========================================================================
NLTK was unable to find the java file!
Use software specific configuration paramaters or set the JAVAHOME environment variable.
===========================================================================
INFO:utils.logger:lexicosyntactic: Done input_data/file1.txt -> output_folder/lexicosyntactic/file1.csv
100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.86s/it]
anandhpe_ualberta_ca@cpu8ram30:~/installs/GitHub/COVFEFE$ cd output_folder/lexicosyntactic/
anandhpe_ualberta_ca@cpu8ram30:~/installs/GitHub/COVFEFE/output_folder/lexicosyntactic$ ls
lu_parses  rst_output  stanford_parses

There is no file1.csv and all other folders are empty

Do you have JAVA_HOME set?
Looks similar to https://github.com/nltk/nltk/issues/1476

Hi @daniyall, Yes

This error is because one of the scripts used by the lex features still uses python2. But it looks like calling python2 is not valid on your system. The immediate fix would be to create a link somewhere in your path that links python2 to python or to change https://github.com/SPOClab-ca/COVFEFE/blob/7ee3233aaf51d5aeea0852fd0f64b77bc6c63bc8/utils/lexicosyntactic/syntactic_features.py#L31 to say python instead of python2.

@daniyall Thanks for the input

Now there is some other issue, what is receptiviti and  lexicosyntactic is supposed to be created as a ouput but then it's throwing error no such file or directory.
ERROR:utils.logger:lexicosyntactic: Failed with error [Errno 2] No such file or directory: '/home/anandhpe_ualberta_ca/installs/GitHub/COVFEFE/output_folder/lexicosyntactic/lu_parses/file1.txt'

You can ignore the receptiviti error. The lexicosyntactic error looks a bit odd. Can you try deleting output_folder and running the pipeline again?

@daniyall sorry for the delay
tried deleting the output folder it didn't help

Is there anything inside the lu_parses folder?

@daniyall  nothing is there inside lu_parser, Nevermind I tried in another server it's working fine.
I guess something is wrong with my env.
Thanks anyway

I have the same problem as @anandhperumal exactly. I also made the change he made, and it caused the same problem for me. In the subprocess.call in syntactic_features.py, it mentions 'python2' and 'analyzeText.py' ... where is the latter coming from? Is it necessary to reference python2? I'm working in a virtual env per the ReadMe, but it's not working.

I believe some of the dependencies require python2 which is why its being called. Are you using Windows, Mac or Linux?

Windows... I have Python 2 installed but I am using a virtual environment bc COVFEFE wants old versions of programs . Not sure if that could raise an issue
Get Outlook for iOShttps://aka.ms/o0ukef

From: daniyall<EMAIL_ADDRESS>Sent: Monday, December 7, 2020 5:23:08 AM
To: SPOClab-ca/COVFEFE<EMAIL_ADDRESS>Cc: kearnsdm<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [SPOClab-ca/COVFEFE] Output Folder is empty (#19)
I believe some of the dependencies require python2 which is why its being called. Are you using Windows, Mac or Linux?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/SPOClab-ca/COVFEFE/issues/19#issuecomment-739823798, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB2536UQ5A5R66VADTFIFLDSTSUIZANCNFSM4JB7F3JA.

It looks like it is just analyzeText that is needed from Python 2. Is there an analogue in Python 3 or another way to handle this?


Support feedback from the runtime

Library can provide some feedback back to app (traces, alerts, events, etc)

Implemented, docs were added in 71ba54fb3


StreamConfigurationProviders cannot be loaded more than once

Describe the bug
Using SpringBootEnvProvider in conjunction with PlaceholderResolver results in an error:
Unhandled exception. System.InvalidOperationException: StreamConfigurationProviders cannot be loaded more than once.
   at Microsoft.Extensions.Configuration.StreamConfigurationProvider.Load()
   at Steeltoe.Extensions.Configuration.SpringBoot.SpringBootEnvProvider.Load()

Steps to reproduce
Steps to reproduce the behavior:
1.AddPlaceholderResolver
2.Use any extension method that adds AddSpringBootFromEnvironmentVariable
Expected behavior
Should work without throwing errors
Environment (please complete the following information):

Steeltoe Version [e.g. 3.2.3]
.NET Version [e.g. .NET 6]
Any other library versions to note

@hananiel does #1170 still need to be applied to main?

Yes this needs to be applied to main as well.


Layout issues in Running Extensions view

Testing #39568

Notice the flex glitches
Notice how the three textual labels are all vertically misaligned
The only UX indication that the three textual labels are actually related to one another becomes evident only when the user hovers a row with the mouse

In the latest Insiders (https://github.com/Microsoft/vscode/commit/12f10aba98bab2f1e5547ba4a5f8fce6d535c0ee):
Report Issue buttom is not aligned when there is a warning/error for the extension or extension name is long

That is fine for now. Since we plan to move it to the extensions viewlet either way


HTML entity codes of Special Text characters getting Currency Symbols

Hello,
How it is Possible to find out the Currency Symbol like,
I mentioned if i am given like .  £(this is HTML entity ,how it is Converted to Currency Symbol like as pound symbol £ ).

@lavarajallu I don't think your question is strictly related to React Native, but if I understand your problem correctly you should be able to use a library like html-entities to decode/encode HTML Character Entities.

This issue looks like a question that would be best asked on StackOverflow.
StackOverflow is amazing for Q&A: it has a reputation system, voting, the ability to mark a question as answered. Because of the reputation system it is likely the community will see and answer your question there. This also helps us use the GitHub bug tracker for bugs only.
Will close this as this is really a question that should be asked on StackOverflow.

@levibuzolic  Thanks a lot.......


[#147669197] Upgrade stemcells for USN-3334-1

What
Update BOSH and Concourse stemcells for https://www.cloudfoundry.org/usn-3334-1/
I've also changed Concourse stemcell series to match what's used by BOSH and CF.
How to review
Code review only. Check that it's the correct stemcell version specified by the USN.
I've tested upgrading from the master branch in dev.
🐝  I'll deploy to the persistent environment after it's merged 🐝
Who can review
Anyone.

🐝  I had a problem with deploying this in dev. But then I rolled back to master and upgraded again, and didn't have any problems. So I think it was unrelated 🐝
Started installing CPI
  Compiling package 'ruby_aws_cpi/5e8696452d4676dd97010e91475e86b23b7e2042'... Finished (00:00:00)
  Compiling package 'bosh_aws_cpi/e211ebbf3d181c107f4e06a58dd45dbf30283bb6'... Finished (00:00:00)
  Installing packages... Finished (00:00:12)
  Rendering job templates... Finished (00:00:10)
  Waiting for the agent on VM 'i-0224c411258ababe4'...  Stopping jobs on instance 'unknown/0'...  Unmounting disk 'vol-0fba0667e550a1523'... Finished (00:00:32)
  Attaching disk 'vol-0fba0667e550a1523' to VM 'i-00f87e635af099898'...  Rendering job templates... Finished (00:00:01)
  Compiling package 'ruby/c1086875b047d112e46756dcb63d8f19e63b3ac4'... Finished (00:01:35)
  Compiling package 'ruby_aws_cpi/5e8696452d4676dd97010e91475e86b23b7e2042'... Finished (00:00:01)
 Finished (00:02:42)
  Compiling package 'registry/7751f332d697c84b5137bdbc7a6cea78fffde75d'... Finished (00:00:01)
  Compiling package 'collectd/e2d748928525612e8bc7f99173dfa9ce96367acb'... Finished (00:00:47)
  Compiling package 'bosh_aws_cpi/e211ebbf3d181c107f4e06a58dd45dbf30283bb6'... Finished (00:00:50)
  Compiling package 'davcli/5f08f8d5ab3addd0e11171f739f072b107b30b8c'... Finished (00:00:01)
  Compiling package 'director/a6bbfcee2c7cf65840ec525b6e3181da6cf59598'... Finished (00:01:03)
  Compiling package 'nginx/2ec2f63293bf6f544e95969bf5e5242bc226a800'... Finished (00:00:13)
  Updating instance 'bosh/0'... Finished (00:00:18) Finished (00:00:08)
  Running the post-start scripts 'bosh/0'... Finished (00:00:05)
Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs...

connection: process error: could not find the exitcode file for the process: stat /worker-state/depot/33bdc073-cb70-4622-6b0e-2fb7ff0d0e3c/processes/47ba2ae8-58cc-4c17-79b9-8ad23b1d35c4/exitcode: no such file or directory


"no connected devices" when we see an unauthorized device

We shouldn't say "no connected devices" if we see an unauthorized device.
$ flutter run
Device 0012bae55d88a1be is not authorized.
You might need to check your device for an authorization dialog.
No connected devices.

What do you think the simplest fix is for this? Based on where this call is printed in flutter_command.dart, other 0-device cases are handled like this:
    } else if (devices.isEmpty && deviceManager.hasSpecifiedAllDevices) {
      printStatus('No devices found');
      return null;
    }

However, I don't currently see anything like deviceManager.devicePendingAuthorization in DeviceManager.
I could see either:

Adding a method like devicePendingAuthorization to DeviceManager.
Simply changing No connected devices to No available devices.

Number 2 seems less thorough but just as accurate. There's no need to add an additional 'authorized device' state if this error message is the only place that needs it, I would argue.
What do you think?

I wouldn't necessarily aim for the simplest fix, but I agree with you that there's nothing really gained from doing anything more involved than just changing the message in this case.

I meant simple in a good way, fwiw.
However, I can't repro the output you show above. Do you mean also for the output of flutter devices, as shown below?
$ flutter --version
Flutter 0.7.1-pre.30 • channel master • git@github.com:srsudar/flutter.git
Framework • revision 7f0f938de7 (12 hours ago) • 2018-08-27 11:11:14 +0200
Engine • revision af42b6dc95
Tools • Dart 2.1.0-dev.1.0.flutter-ccb16f7282

$ flutter run
No connected devices.

$ flutter devices
No devices detected.

Run 'flutter emulators' to list and start any available device emulators.

Or, if you expected your device to be detected, please run "flutter doctor" to diagnose
potential issues, or visit https://flutter.io/setup/ for troubleshooting tips.

• Device 01abfc49119c410e is not authorized.
  You might need to check your device for an authorization dialog.

It's possible this has improved since the original bug was filed.

we met the  same question,  Have you solved it?


Set .symmetric_difference() Operation

Solved and Added to the Repo : https://github.com/jarvismayur/Hackerrank-Problem-Solutions/blob/main/Python/Set .symmetric_difference() Operation/Set_.symmetric_difference()_Operation.py


Notice regarding AuroraProxy Documentation

As this project advances, we will move documentation from README.md to a github wiki where topics will be explained in detail

cool

HIII GUYSS

how to clone the respository and start the server in bash?

rip aurora, development is severely limited or the proxy is down.

Aurora dead


[core-http] Throttling retry policy fix in core-http

Closing in favour of https://github.com/Azure/azure-sdk-for-js/pull/15832


Two dates in .expiration_date

Can anyone tell me why sometimes the expiration_date brings me a list with two dates (examples: google.com, amazon.com, paypal.com, microsoft.com) and sometimes brings me only one datetime?
Thanks! :-)

Because WHOIS server returns two different timestamps. Please see:
$ whois google.com
   Domain Name: GOOGLE.COM
   Registry Domain ID: 2138514_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.markmonitor.com
   Registrar URL: http://www.markmonitor.com
   Updated Date: 2019-09-09T15:39:04Z
   Creation Date: 1997-09-15T04:00:00Z
   Registry Expiry Date: 2028-09-14T04:00:00Z
   Registrar: MarkMonitor Inc.
   Registrar IANA ID: 292
   Registrar Abuse Contact Email<EMAIL_ADDRESS>   Registrar Abuse Contact Phone: +1.2083895740
   Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
   Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited
   Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited
   Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited
   Name Server: NS1.GOOGLE.COM
   Name Server: NS2.GOOGLE.COM
   Name Server: NS3.GOOGLE.COM
   Name Server: NS4.GOOGLE.COM
   DNSSEC: unsigned
   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2021-02-27T08:25:04Z <<<

For more information on Whois status codes, please visit https://icann.org/epp

NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar.  Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.

TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability.  VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.

The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.
Domain Name: google.com
Registry Domain ID: 2138514_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2019-09-09T08:39:04-0700
Creation Date: 1997-09-15T00:00:00-0700
Registrar Registration Expiration Date: 2028-09-13T00:00:00-0700
Registrar: MarkMonitor, Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email<EMAIL_ADDRESS>Registrar Abuse Contact Phone: +1.2083895770
Domain Status: clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)
Domain Status: clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)
Domain Status: clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)
Domain Status: serverUpdateProhibited (https://www.icann.org/epp#serverUpdateProhibited)
Domain Status: serverTransferProhibited (https://www.icann.org/epp#serverTransferProhibited)
Domain Status: serverDeleteProhibited (https://www.icann.org/epp#serverDeleteProhibited)
Registrant Organization: Google LLC
Registrant State/Province: CA
Registrant Country: US
Registrant Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com
Admin Organization: Google LLC
Admin State/Province: CA
Admin Country: US
Admin Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com
Tech Organization: Google LLC
Tech State/Province: CA
Tech Country: US
Tech Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com
Name Server: ns2.google.com
Name Server: ns1.google.com
Name Server: ns3.google.com
Name Server: ns4.google.com
DNSSEC: unsigned
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2021-02-27T00:19:42-0800 <<<

For more information on WHOIS status codes, please visit:
  https://www.icann.org/resources/pages/epp-status-codes

If you wish to contact this domain’s Registrant, Administrative, or Technical
contact, and such email address is not visible above, you may do so via our web
form, pursuant to ICANN’s Temporary Specification. To verify that you are not a
robot, please enter your email address to receive a link to a page that
facilitates email communication with the relevant contact(s).

Web-based WHOIS:
  https://domains.markmonitor.com/whois

If you have a legitimate interest in viewing the non-public WHOIS details, send
your request and the reasons for your request to<EMAIL_ADDRESS>and specify the domain name in the subject line. We will review that request and
may ask for supporting documentation and explanation.

The data in MarkMonitor’s WHOIS database is provided for information purposes,
and to assist persons in obtaining information about or related to a domain
name’s registration record. While MarkMonitor believes the data to be accurate,
the data is provided "as is" with no guarantee or warranties regarding its
accuracy.

By submitting a WHOIS query, you agree that you will use this data only for
lawful purposes and that, under no circumstances will you use this data to:
  (1) allow, enable, or otherwise support the transmission by email, telephone,
or facsimile of mass, unsolicited, commercial advertising, or spam; or
  (2) enable high volume, automated, or electronic processes that send queries,
data, or email to MarkMonitor (or its systems) or the domain name contacts (or
its systems).

MarkMonitor reserves the right to modify these terms at any time.

By submitting this query, you agree to abide by this policy.

MarkMonitor Domain Management(TM)
Protecting companies and consumers in a digital world.

Visit MarkMonitor at https://www.markmonitor.com
Contact us at +1.8007459229
In Europe, at +44.02032062220
--

Notice:
   Registry Expiry Date: 2028-09-14T04:00:00Z

And:
Registrar Registration Expiration Date: 2028-09-13T00:00:00-0700

Please also read this part:
NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar.  Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.


Added Fedora and CentOS Preview 2 links to .NET Core Linux Prerequisites

Added the just released Fedora and CentOS Preview 2 links to .NET Core Linux Prerequisites.
Internal Review URL

Thanks, @mairaw!


modules:  Migrate Glide => Go 1.11 modules, de-vendorize

Supporting only Go 1.11 with this change, to benefit from build reproducibility of modules with reduced need for vendoring. Taking my chances without it.
In Go 1.9 they stopped including vendor in ./... for many tools, allowing a few Makefile hacks to be removed.
Let's see what I break on Travis with this…

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@3591fbe). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #4   +/-   ##
=========================================
  Coverage          ?   87.39%
=========================================
  Files             ?        3
  Lines             ?      119
  Branches          ?        0
=========================================
  Hits              ?      104
  Misses            ?        9
  Partials          ?        6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3591fbe...bf8173e. Read the comment docs.


ROS2 Support?

Will this package be usable for ROS2? I am currently looking into making a generic subscriber and then unpack the values into a JSON, but have no found if something like this is possible in ROS2. Any ideas?

I implemented this already in PlotJuggler, but it is not currently available as an isolated package. It will eventually

Thank you for your reply. I found your package ros2_introspection, but am confused as to how to obtain  rmw_serialized_message_t* serialized_message;. Would you be so kind as to explain this to me?
To make it scalable, I presume you need a string name of all topic's message_type (where you need to register once). Have you implemented something like this before?
Many thanks

I presume the serialized_message comes from https://github.com/ros2/rosbag2/blob/c34601acba77d08c22211d3eb062b8ba867fef71/rosbag2_transport/src/rosbag2_transport/generic_subscription.cpp?

exactly, this means that it change in all the version of ROS2 (API is continuously broken)

Thank you! I will try to take it from here, might pop back in case I'm really stuck

please use the Github issue of ros2_introspection, this repo is related to ROS1 only


Panic when start cyfs-meta-genesis in Debug

thread 'main' has overflowed its stack

The closure in async_std::task::block_on is too large for stack, put it in heap with Box.

This phenomenon is easy to occur when running services and tools in debug mode.
In most cases, you need to compile the release version and then run itIn most cases, you need to compile the release version and then run it

This phenomenon is easy to occur when running services and tools in debug mode.
In most cases, you need to compile the release version and then run itIn most cases, you need to compile the release version and then run it

Yes, I've found some solutions:

Move the large closure in heap from stack.
Cut the large function into little. But I don't known how it works, anyone tell me?

This phenomenon is easy to occur when running services and tools in debug mode.
In most cases, you need to compile the release version and then run itIn most cases, you need to compile the release version and then run it

Yes, I've found some solutions:

Move the large closure in heap from stack.
Cut the large function into little. But I don't known how it works, anyone tell me?

It seems that we can approach this from two perspectives:

Reduce the size of the data stored on the stack, for example by using heap instead of stack.
Switch threads, especially for some deep calls. If it can be completed among multiple threads, it should be better, such as using spawn_task.

Unless debugging is needed in the debug version, it can be run directly in release mode to avoid this.


⚠ Activitar Portal has degraded performance

In b005371, Activitar Portal (https://activitar.com) experienced degraded performance:

HTTP code: 200
Response time: 4381 ms

Resolved: Activitar Portal performance has improved in 6320d32.


feat(keycloak): add seeding data to configuration

Description
The keycloak-seeder has been changed to (optionaly) read seeding-data from it's settings and merge this additional data with the one that is read from json before writing changes to keycloak. This allows to override values in the keycloak-json by settings being passed as environment-variables (which can be defined via helm-chart).
Why
seeding e.g. secrets from json-files would uncover confidential data. It also is unflexible - settings defined by environment-variables can e.g. be defined in kubernetes secrets.
Issue
#438
Checklist
Please delete options that are not relevant.

[X] I have followed the contributing guidelines
[X] I have performed a self-review of my own code
[X] I have successfully tested my changes locally
[X] I have added tests that prove my changes work
[X] I have checked that new and existing tests pass locally with my changes
[X] I have commented my code, particularly in hard-to-understand areas

Hi @ntruchsess I just tried to import this realm from scratch (not only at upgrade / updating existing realm configuration) and I'm encountering the following error:
{"Timestamp":"2024-06-24T16:12:24.1533063+00:00","Level":"Information","MessageTemplate":"Start seeding","RenderedMessage":"Start seeding"}
{"Timestamp":"2024-06-24T16:12:25.0499479+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 404 (Not Found): GET http://centralidp-keycloak/auth/admin/realms/CX-Central\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7938875+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7992279+00:00","Level":"Fatal","MessageTemplate":"Unhandled exception","RenderedMessage":"Unhandled exception","Exception":"System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   --- End of inner exception stack trace ---\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51\n   at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)\n   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58\n   at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86"}
Unhandled exception. System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
 ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   --- End of inner exception stack trace ---
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51
   at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)
   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58
   at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86
   at Program.<Main>(String[] args)

The issue is not specific to your change as I tested it also with the previous state but it's relevant now because we want to use the job to setup entire realms not just updating existing ones.

Hi @ntruchsess I just tried to import this realm from scratch (not only at upgrade / updating existing realm configuration) and I'm encountering the following error:
{"Timestamp":"2024-06-24T16:12:24.1533063+00:00","Level":"Information","MessageTemplate":"Start seeding","RenderedMessage":"Start seeding"}
{"Timestamp":"2024-06-24T16:12:25.0499479+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 404 (Not Found): GET http://centralidp-keycloak/auth/admin/realms/CX-Central\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7938875+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7992279+00:00","Level":"Fatal","MessageTemplate":"Unhandled exception","RenderedMessage":"Unhandled exception","Exception":"System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   --- End of inner exception stack trace ---\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51\n   at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)\n   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58\n   at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86"}
Unhandled exception. System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
 ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   --- End of inner exception stack trace ---
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51
   at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)
   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58
   at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86
   at Program.<Main>(String[] args)

The issue is not specific to your change as I tested it also with the previous state but it's relevant now because we want to use the job to setup entire realms not just updating existing ones.

I guess (at least one) role that keycloak creates by default is missing in the json. Whenever a realm is being created keycloak implicitly creates roles to manage administrative privileges of that realm. Those realm-management-roles cannot be deleted. What happens is that the seeding-job creates the realm, keycloak creates the management-roles, the seeding-job now queries the realm and calculates the delta to what is defined in json. Whenever a role is existing in keycloak that is not also defined in json the seeding-job tries to delete it. Unfortunally those realm-management-roles are not marked as such so the seeder cannot destinguish in between manually and implicitly created roles. Therefore for a successful import all implicitly created roles must also be defined in the json.

could you please also change / rebase the pull request to main, this change isn't meant for our 2.1.0 version

done

Hi @ntruchsess I just tried to import this realm from scratch (not only at upgrade / updating existing realm configuration) and I'm encountering the following error:
{"Timestamp":"2024-06-24T16:12:24.1533063+00:00","Level":"Information","MessageTemplate":"Start seeding","RenderedMessage":"Start seeding"}
{"Timestamp":"2024-06-24T16:12:25.0499479+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 404 (Not Found): GET http://centralidp-keycloak/auth/admin/realms/CX-Central\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7938875+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7992279+00:00","Level":"Fatal","MessageTemplate":"Unhandled exception","RenderedMessage":"Unhandled exception","Exception":"System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   --- End of inner exception stack trace ---\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51\n   at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)\n   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71\n   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58\n   at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86"}
Unhandled exception. System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
 ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   --- End of inner exception stack trace ---
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51
   at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)
   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71
   at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58
   at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86
   at Program.<Main>(String[] args)

The issue is not specific to your change as I tested it also with the previous state but it's relevant now because we want to use the job to setup entire realms not just updating existing ones.

I guess (at least one) role that keycloak creates by default is missing in the json. Whenever a realm is being created keycloak implicitly creates roles to manage administrative privileges of that realm. Those realm-management-roles cannot be deleted. What happens is that the seeding-job creates the realm, keycloak creates the management-roles, the seeding-job now queries the realm and calculates the delta to what is defined in json. Whenever a role is existing in keycloak that is not also defined in json the seeding-job tries to delete it. Unfortunally those realm-management-roles are not marked as such so the seeder cannot destinguish in between manually and implicitly created roles. Therefore for a successful import all implicitly created roles must also be defined in the json.

As discussed, renaming the default role to the default pattern did solve the issue

I noticed that seeding of the (bpn) user attribute via seeding config fails (with error, see logs-from-centralidp-keycloak-realm-seeding-in-centralidp-keycloak-realm-seeding-1-5qw8s.log) for service account users:
used config:
          - name: "KEYCLOAKSEEDING__REALMS__0__USERS__0__Attributes__0__NAME"
            value: "bpn"
          - name: KEYCLOAKSEEDING__REALMS__0__USERS__1__USERNAME
            value: "service-account-sa-test-2"
          - name: KEYCLOAKSEEDING__REALMS__0__USERS__1__Attributes__0__VALUES__0
            value: TEST2

I tested with a normal user as well, there it worked.

this is missing:
- name: KEYCLOAKSEEDING__REALMS__0__USERS__1__Attributes__0__NAME

Need to revert the approval as c5b4a10 is causing this error logs-from-centralidp-keycloak-realm-seeding-in-centralidp-keycloak-realm-seeding-1-wvnxh.log

As discussed - this error was due to invalid configuration that was not compliant with the startup-configuration-checks.

Please wait for my approval before merge, I hope I get the final tests soon


训练transmitter时报错，TypeError: 'NoneType' object is not callable

The following parameters will be optimized NORMALLY:
linear.weight
[ training... ]
[ Saving tensorboard logs here: ./tmp/transmitter/tensorboard ]
preinitializing pytorch cuda buffer
Traceback (most recent call last):
File "train_transmitter.py", line 145, in
TrainLoop(opt).train()
File "/disk1/lenny/share/PersonaDG/Persona-Dialogue-Generation/parlai/scripts/train_model.py", line 512, in train
world.parley()
File "/disk1/lenny/share/PersonaDG/Persona-Dialogue-Generation/parlai/core/worlds.py", line 663, in parley
batch_act = self.batch_act(agent_idx, batch_observations[agent_idx])
File "/disk1/lenny/share/PersonaDG/Persona-Dialogue-Generation/parlai/core/worlds.py", line 636, in batch_act
batch_actions = a.batch_act(batch_observation)
File "/disk1/lenny/share/PersonaDG/Persona-Dialogue-Generation/agents/transmitter/transmitter.py", line 857, in batch_act
self.init_cuda_buffer(batchsize)
File "/disk1/lenny/share/PersonaDG/Persona-Dialogue-Generation/agents/transmitter/transmitter.py", line 842, in init_cuda_buffer
sc = self.model(input_dummy, None, None, output_dummy, None)[1]
File "/home/lenny/anaconda3/envs/t18p20/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/disk1/lenny/share/PersonaDG/Persona-Dialogue-Generation/agents/transmitter/gpt/model.py", line 91, in forward
lm_logits, hidden_states = self.transformer_module(input_seq, None, dis_seq)
TypeError: 'NoneType' object is not callable

@1973sir  I cannot reproduce the errors. I think the installation of transformers library is not correct (we have customized some parts to support the auxiliary task).
Please see here to see how to install the correct dependencies. Hope it helps you.

after create a new env with torch 1.01 and cuda 10.0 ,this error disapears.


#1912 Use top-level statements for Program in samples and update Getting Started docs

Closes #1912

#1912

Proposed Changes

Enhanced Ocelot logger: improved log readability including console, resolved triple logging of Exception objects
Refactored Basic sample
❗This is the key change due to the issue requirements
Created new basic Configuration sample
Updated gettingstarted.rst, rewritten Getting Started documentation
❗This is the key change due to the issue requirements
Updated all samples with top-level statements for Program.cs and improved project structure based on modern .NET 8 web app templates

@RaynaldM @ggnaegi Where are you both, guys?


flutter doctor --android-licenses returns exceptions in Android Studio Arctic Fox

Hello,
I have just updated my Android Studio to latest version Arctic Fox.
I have checked for updates for Flutter plugin, but was already at the latest.
Now I run flutter doctor from terminal, to check everything is OK :
Microsoft Windows [Version 10.0.19042.1237]
(c) Microsoft Corporation. All rights reserved.
C:>flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, 2.0.5, on Microsoft Windows [Version 10.0.19042.1237], locale en-GB)
[!] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
X Android license status unknown.
Run flutter doctor --android-licenses to accept the SDK licenses.
See https://flutter.dev/docs/get-started/install/windows#android-setup for more details.
[√] Chrome - develop for the web
[√] Android Studio (version 4.1.0)
[√] Connected device (2 available)
! Doctor found issues in 1 category.
C:>flutter doctor --android-licenses
Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema
at com.android.repository.api.SchemaModule$SchemaModuleVersion.(SchemaModule.java:156)
at com.android.repository.api.SchemaModule.(SchemaModule.java:75)
at com.android.sdklib.repository.AndroidSdkHandler.(AndroidSdkHandler.java:81)
at com.android.sdklib.tool.sdkmanager.SdkManagerCli.main(SdkManagerCli.java:73)
at com.android.sdklib.tool.sdkmanager.SdkManagerCli.main(SdkManagerCli.java:48)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlSchema
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 5 more
what could be causing this issue ?
thank you

Hey @santaanto because of the update, your machine needs to ad new sdk tools in Android Studio, you can checkout other responses here, this is a similar issue https://github.com/flutter/flutter/issues/56778

Thank you, I had just fixed this by selecting "Android SDK command line tools (latest)" from Tools->Sdk Manager->SDK Tools menu. I don't undertand why it was unselected by the new update as with previous installation this was OK, but anyway now it works ! thank you


fix: upgrade to version 3.0.526

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign jstrachan
You can assign the PR to them by writing /assign @jstrachan in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jenkins-x-bot-test: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gsm
2132e5b157fdd9987d8b96a5305218246b2c7315
link
/test gsm

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

@jenkins-x-bot-test: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gsm
2132e5b157fdd9987d8b96a5305218246b2c7315
link
/test gsm

vault
2132e5b157fdd9987d8b96a5305218246b2c7315
link
/test vault

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.


location not showing

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Provide a piece of code to reproduce the problem.
Traceback
Show your full traceback so that it is clear where exactly the error occurred.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. Ubuntu 21.04]
Python version
instagrapi version
moveipy version if used
imagemagick version if used

Additional context
Add any other context about the problem here.

@dheeraj009joshi you did not fill in any of the fields, I am waiting for you to provide data (source code to reproduce, traceback, screenshots, additional context such as region) to solve the problem


Fix obvious typo. SWIFT_BULID_STDLIB => SWIFT_BUILD_STDLIB.

I think we didn't catch this since we also check if overlays are enabled here
and generally overlays and stdlib are enabled together. It is NFC from the
actual build's perspective since this just controls whether or not diagnostic
information is printed out about what SDKs/etc is configured to be built.

@swift-ci smoke test and merge

@swift-ci clean smoke test linux platform

Hmmm... I don't think that linux failure is me.

@swift-ci smoke test linux platform

@swift-ci clean smoke test linux platform


docs: add list-like to table of parameters for all backends

Add list_like parameter to the table of supported configuration parameters, for all backends.

Codecov Report
Base: 88.13% // Head: 41.81% // Decreases project coverage by -46.32% :warning:

Coverage data is based on head (9d04395) compared to base (efc203f).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #768       +/-   ##
===========================================
- Coverage   88.13%   41.81%   -46.33%
===========================================
  Files         133      133
  Lines        6703     6703
===========================================
- Hits         5908     2803     -3105
- Misses        795     3900     +3105

Flag
Coverage Δ

docarray
41.81% <ø> (-46.33%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

 Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Proposal: Show overall number of modules like Deno is doing - see screenshot

in gallery.
Just exploring how this goes together with the lazy loading 24 ... + more on demand ...
If somebody reading this knows if the API provides something like https://x.nest.land/api/modulecount/ please let me know.

It is provided as a HTTP response header:
https://github.com/nestdotland/x/blob/c94add3607c9fb7b45ac916d5f0776eb0fc0aeb1/src/routes/package.ts#L44

cool thanks - will probably draft this tomorrow.

Would love to see this added! Really smart thinking, @TTtie.

Perhaps we could even consider showing it beside the "shown modules" count instead...

Perhaps we could even consider showing it beside the "shown modules" count instead...

I'd rather get rid of this one (x Modules Shown) - it is over the Deno's foot traces - and the info (that there are many modules...) is pretty well in the search field (Search through x modules).

It is provided as a HTTP response header:
https://github.com/nestdotland/x/blob/c94add3607c9fb7b45ac916d5f0776eb0fc0aeb1/src/routes/package.ts#L44

It's really cool to have X-Total however it seems that the following is necessary to enable this:
Access-Control-Expose-Headers: X-Total
Background:
https://stackoverflow.com/questions/37897523/axios-get-access-to-response-header-fields
If that's right: Could anyone tell me where I can do that because so far response.headers only contains / reveals content-type when called via browser / page ?

Perhaps we could even consider showing it beside the "shown modules" count instead...

Maybe something like "Showing 24 of 100 modules"?

CC @maximousblk & @justablob

I like @TheWizardBear's idea of showing

Showing $visible of $total modules

I'd even suggest we should do what deno has with the search box

I really like @TheWizardBear's idea too. I'd be happy with either representation of this data.

ok - does anyone have updates on this one:
https://github.com/nestdotland/nest.land/issues/267#issuecomment-691703533
As this is imo necessary to get the X-Total


TernarySearch.py

fix #104

Hello @MadJokkerr, we appreciate your contribution. However, we noticed that this issue has been assigned to someone else, and not to you. You have created a pull request without being assigned an issue, this goes against our repository guidelines.
Please look for an issue that interests you, or create a new one.
Have a good day!


adjusting the default help behaviour

"-h"
The docs state just
kingpin.CommandLine.HelpFlag.Short('h')

but I am not quite sure what to do with it.
Just calling it does not seem to work.
Would be great to expand the docs on that.
always use "--help-long"
Is there a way to always print "--help-long" also for "--help"?
I am not a fan of having two separate version for the help.

Just calling it does not seem to work.

Calling .Short('h') adds -h as a short option to the help flag. To clarify, what were you expecting it to do?

Is there a way to always print "--help-long" also for "--help"?

To change the default help template you should use UsageTemplate(template), passing one of the predefined templates (LongHelpTemplate in your case).

Customising help output is described here.

@alecthomas the .Short('h') in theory is obvious - but the "help" flag is generated automatically.
The docs just say kingpin.CommandLine.HelpFlag.Short('h') and it's not exactly clear how that is meant to be used with e.g.
  var (
    app      = kingpin.New("foo", "bad")
    debug    = app.Flag("debug", "bla").Short('d').Bool()
  )

Looking at the code it becomes clear that
app.HelpFlag.Short('h')

does the trick. So it would be great to improve the docs.
Thanks for the help with the template. I didn't realize this is due to switching templates. And indeed app.UsageTemplate(kingpin.LongHelpTemplate) does the trick.
The only thing that still does not make much sense is the (also try --help-long and --help-man). Looking at the code I see in app.go
	a.HelpFlag = a.Flag("help", "Show context-sensitive help (also try --help-long and --help-man).")
	a.HelpFlag.Bool()
	a.Flag("help-long", "Generate long help.").Hidden().PreAction(a.generateLongHelp).Bool()
	a.Flag("help-man", "Generate a man page.").Hidden().PreAction(a.generateManPage).Bool()

which is a bit too static for my taste.
I don't want a Flag("help-long") at all - and I would love to get the (also try --help-long and --help-man) out of the description.

I don't want a Flag("help-long") at all - and I would love to get the (also try --help-long and --help-man) out of the description.

This is fixed in v3-unstable.

Thanks!


chore: add commented module addition for user

required for e2e test, it will uncomment the line to create a new user.
Signed-off-by: João Vanzuita<EMAIL_ADDRESS>

please, also update the docs: https://docs.kubefirst.io/common/terraform.html#how-can-i-use-atlantis-to-add-a-new-user-on-my-github-backed-installation

done at https://github.com/kubefirst/kubefirst/pull/1234


Navigation loads slowly

most likely some blocks take long to load/render, and the header is only rendered once all the blocks returned their async processing.

not not reproduce on .live


Save state when updating code

Is it possible when an update is uploaded, that the previous auto/manual saved settings can be loaded, either automaticslly or manually?
Each time time I set my display up the way I like/need it and other configertions, after a new code is uploaded all previous settings are gone.  I do find this part annoying. I assume the saved settings are stored on internal flash and gets wiped at each new upload? If so, is it possible to save settings to SDCARD, thus allowing easy application of settings UNLESS features change considerably that the saved state causes corruption  to non-existent or changed menus options? I'm not sure on the logical way to tackle this, yet.

Just to clarify.

Save your settings
Load new firmwary
Load your settings
You should then make sure they are correct, including any new ones that have been added, and then save again.

Thanks for the insight and clarification. My step was to first turn on AUTOLOAD SETTINGS and I think, it would do an autosave of current state, thus overwriting my previous saved settings?
As, after doing that and THEN trying to load previous saved settings, there was nothing.
I will need to do as you said, THEN turn on AUTOLOAD SETTINGS.
I'll try for next version of code, as I've already applied your latest, before seeing this message. Thanks

I didn't think it was a good idea to do a save when autoload was turned on, that would overwrite an existing settings file. I suppose I could have designed it to do an autosave if no saved file was saved, but I hadn't thought of that at the time. I think about changing that, I don't think it will cause any problems. However, turning autosave on will still use an existing settings file. I suppose I could ask if the current settings should be save. But then again, hmmm, things get complicated!


db._query(...).then is not a function

I'm trying to run a query within a transaction and use the results of the query to manipulate some of the data.
The documentation on using transactions within arangojs is basically non-existent, but i've managed to get this far and now i'm getting the error db._query(...).then is not a function with the transaction code below.
How can I get the results of my query to manipulate them?
const query = String(function (params) {
      const db = require('@arangodb').db

      const query = `some query which i know works`

          db._query(query)
            .then(cursor => {
              return cursor.next().then(result => {
                if (result !== undefined) {
                  // do stuff
                }
              })
            })
            .catch(err => {
              throw err
            })

transaction =
      db.transaction({
        write: ['mycollection'],
        read: ['mycollection']
      },
      query,
      {})

The code inside the action function is executed in ArangoDB, not in Node, which is why the API is also slightly different:
https://docs.arangodb.com/3.3/Manual/Appendix/JavaScriptModules/ArangoDB.html
Note that among other things unlike arangojs every ArangoDB API is synchronous. So the reason the then isn't working for you is that it's not returning a promise.

Excellent i didn't know that. Is it documented anywhere that ArangoDB is synchronous?
IMO the ArangoDB examples are lacking. Some on transactions, these synchronous returns, etc would be helpful.

I agree that the documentation of the ArangoDB JS API is a bit lacking. The primary use cases covered by that documentation right now are Foxx and arangosh. If you can point out any particular things you'd like us to improve, please open a ticket with the ArangoDB main repo

Added a link to this issue in an internal planning ticket:
"Docs: Distinguish clearer between ArangoJS and arangodb module"


[DEVR-97] Team Invitation Flow

Changes

Added team invitation flow
Moved team invitations model to team collaborator
Added mechanism to join a team when there's an invitation

This is


choreupgrade spring to 5.3.39 (#39863)

Proposed changes
Issue Number: close #xxx

(cherry picked from commit a692af226a8292d1b0a409c914d616157b64c82a)
Proposed changes
Issue Number: close #xxx

run buildall


Create a test script that exercises all leg servos

Create a script that moves each servo in each leg in a slow systematic way to allow easy observation.
This will allow us to confirm that both robots are wired the same way, and will also act as a start up test to detect the case where one or more servos aren't responding.

the attached compiler script and it's compiled version (for Doug's bot ID) does the following for each leg in succession:
-raises the toe 3 cm
-moves the toe away from the chassis by 3 cm (sort of)
-moves the tow 3 cm towards the front of the robot (sort of)
These moves are all done via local offsets, and they each exercise only one servo motor. So for the corner legs, the second stage above moves the leg along the local X axis, which is at 45 degrees to the global X and Y axes
Similarly, for the 3rd stage above the toes move along the local Z axis, which is at 45 degrees to the global X and Y axes
testAllLegs.txt
01_hb-testAllLegs.txt
github doesn't like .js files, so I renamed the compiled version to .txt
This script uses the new MoveRelLastLocal script command, which won't be available until I do a git merge. (hopefully in next 20 minutes)


feat: add smart pointer concept

This is long, but I think it is pretty straightforward. Maybe I have tunnel vision and this is not readable at all :D

About.md will follow when the main text has been reviewed.

The only think I'm missing is the reference to works of fiction.

Apart from king Arthur there is also "The Expanse" in there.