Cilium Ingress Controller responds with "403 Access denied" in policyAuditMode=true after upgrade to cilium v1.14.5

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
After upgrading cilium from v1.14.3 to v1.14.5, requests to the cilium ingress controller are dropped with
Dec 30 09:32:45.423: <IP_ADDRESS>:58682 (ingress) -> kube-system/cilium-ingress:80 (world) http-request DROPPED (HTTP/1.1 GET http://podinfo.my.domain/)
Dec 30 09:32:45.423: <IP_ADDRESS>:58682 (ingress) <- kube-system/cilium-ingress:80 (world) http-response FORWARDED (HTTP/1.1 403 0ms (GET http://podinfo.my.domain/))

as soon as the endpoint reserved:ingress switches from "default allow" to "default deny" even though cilium runs in policy audit mode (helm value: policyAuditMode=true).
Before I've started to test with policyAuditMode=true, I've set up cilium to run with
policyEnforcementMode: always
hostFirewall:
  enabled: true
extraConfig:
  allow-localhost: policy

and proper CiliumNetworkPolicies and CiliumClusterwideNetworkPolicies in place (ingress working as expected in v1.14.3 and v1.14.4)
Maybe the issue is related to the changes from https://github.com/cilium/cilium/pull/28126
Cilium Version
1.14.5
Kernel Version
Linux talos-worker-1 6.1.61-talos #1 SMP Wed Nov  8 10:43:23 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Kubernetes Version
v1.26.4
Sysdump
No response
Relevant log output
No response
Anything else?
I've used this policy to switch the ingress endpoint to default deny:
---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "allow-icmp-echo-requests"
specs:
  - endpointSelector:
      matchLabels: {}
    ingress:
      - icmps:
          - fields:
              - type: 8
                family: IPv4
              - type: 128
                family: IPv6
    egress:
      - icmps:
          - fields:
              - type: 8
                family: IPv4
              - type: 128
                family: IPv6
  - nodeSelector:
      matchLabels: {}
    ingress:
      - icmps:
          - fields:
              - type: 8
                family: IPv4
              - type: 128
                family: IPv6
    egress:
      - icmps:
          - fields:
              - type: 8
                family: IPv4
              - type: 128
                family: IPv6

The endpoint id of the reserved:ingress endpoint is 519.
I've obtained endpoint details with cilium endpoint get 519 (inside the cilium-agent container) before the automatic switch of the policy enforcement mode to default deny:

Show/Hide
[
{
  "id": 519,
  "spec": {
    "label-configuration": {},
    "options": {
      "ConntrackAccounting": "Enabled",
      "ConntrackLocal": "Disabled",
      "Debug": "Disabled",
      "DebugLB": "Disabled",
      "DebugPolicy": "Disabled",
      "DropNotification": "Enabled",
      "MonitorAggregationLevel": "Medium",
      "PolicyAuditMode": "Enabled",
      "PolicyVerdictNotification": "Enabled",
      "SourceIPVerification": "Enabled",
      "TraceNotification": "Enabled"
    }
  },
  "status": {
    "controllers": [
      {
        "configuration": {
          "error-retry": true,
          "error-retry-base": "2s"
        },
        "name": "endpoint-519-regeneration-recovery",
        "status": {
          "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
          "last-success-timestamp": "0001-01-01T00:00:00.000Z"
        },
        "uuid": "e2b2e633-d5c3-4c2a-9d80-f37947d7d73b"
      },
      {
        "configuration": {
          "error-retry": true,
          "interval": "5m0s"
        },
        "name": "resolve-identity-519",
        "status": {
          "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
          "last-success-timestamp": "2023-12-30T08:02:33.408Z",
          "success-count": 98
        },
        "uuid": "cbeff185-e3a8-4e3b-869c-e26b26191a88"
      },
      {
        "configuration": {
          "error-retry": true,
          "interval": "10s"
        },
        "name": "sync-to-k8s-ciliumendpoint (519)",
        "status": {
          "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
          "last-success-timestamp": "2023-12-30T08:04:35.575Z",
          "success-count": 2924
        },
        "uuid": "37d3cbe5-989b-4d5f-9e03-fd5dfe8e3fb2"
      }
    ],
    "external-identifiers": {
      "pod-name": "/"
    },
    "health": {
      "bpf": "OK",
      "connected": true,
      "overallHealth": "OK",
      "policy": "OK"
    },
    "identity": {
      "id": 8,
      "labels": [
        "reserved:ingress"
      ]
    },
    "labels": {
      "derived": [
        "reserved:ingress"
      ],
      "realized": {},
      "security-relevant": [
        "reserved:ingress"
      ]
    },
    "log": [
      {
        "code": "OK",
        "message": "Successfully regenerated endpoint program (Reason: policy rules added)",
        "state": "ready",
        "timestamp": "2023-12-30T07:58:20Z"
      }
    ],
    "networking": {
      "addressing": [
        {
          "ipv4": "<IP_ADDRESS>"
        }
      ]
    },
    "policy": {
      "proxy-statistics": [],
      "realized": {
        "allowed-egress-identities": [
          0
        ],
        "allowed-ingress-identities": [
          0
        ],
        "build": 67,
        "denied-egress-identities": null,
        "denied-ingress-identities": null,
        "id": 8,
        "l4": {
          "egress": [],
          "ingress": []
        },
        "policy-enabled": "none",
        "policy-revision": 67
      },
      "spec": {
        "allowed-egress-identities": [
          0
        ],
        "allowed-ingress-identities": [
          0
        ],
        "build": 67,
        "denied-egress-identities": null,
        "denied-ingress-identities": null,
        "id": 8,
        "l4": {
          "egress": [],
          "ingress": []
        },
        "policy-enabled": "none",
        "policy-revision": 67
      }
    },
    "realized": {
      "label-configuration": {},
      "options": {
        "ConntrackAccounting": "Enabled",
        "ConntrackLocal": "Disabled",
        "Debug": "Disabled",
        "DebugLB": "Disabled",
        "DebugPolicy": "Disabled",
        "DropNotification": "Enabled",
        "MonitorAggregationLevel": "Medium",
        "PolicyAuditMode": "Enabled",
        "PolicyVerdictNotification": "Enabled",
        "SourceIPVerification": "Enabled",
        "TraceNotification": "Enabled"
      }
    },
    "state": "ready"
  }
}
]

And then after the switch to default deny, I've tried to obtain endpoint details once more, but I got an error:
Error: Failed to decode nested JSON: invalid character 'M' looking for beginning of object key string (
{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}
)

So, I had to use curl to obtain endpoint details with curl --unix-socket /var/run/cilium/cilium.sock http://localhost/v1/endpoint/519:

Show/Hide
{
"id": 519,
"spec": {
  "label-configuration": {},
  "options": {
    "ConntrackAccounting": "Enabled",
    "ConntrackLocal": "Disabled",
    "Debug": "Disabled",
    "DebugLB": "Disabled",
    "DebugPolicy": "Disabled",
    "DropNotification": "Enabled",
    "MonitorAggregationLevel": "Medium",
    "PolicyAuditMode": "Enabled",
    "PolicyVerdictNotification": "Enabled",
    "SourceIPVerification": "Enabled",
    "TraceNotification": "Enabled"
  }
},
"status": {
  "controllers": [
    {
      "configuration": {
        "error-retry": true,
        "error-retry-base": "2s"
      },
      "name": "endpoint-519-regeneration-recovery",
      "status": {
        "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
        "last-success-timestamp": "0001-01-01T00:00:00.000Z"
      },
      "uuid": "e2b2e633-d5c3-4c2a-9d80-f37947d7d73b"
    },
    {
      "configuration": {
        "error-retry": true,
        "interval": "5m0s"
      },
      "name": "resolve-identity-519",
      "status": {
        "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
        "last-success-timestamp": "2023-12-30T09:22:33.422Z",
        "success-count": 114
      },
      "uuid": "cbeff185-e3a8-4e3b-869c-e26b26191a88"
    },
    {
      "configuration": {
        "error-retry": true,
        "interval": "10s"
      },
      "name": "sync-to-k8s-ciliumendpoint (519)",
      "status": {
        "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
        "last-success-timestamp": "2023-12-30T09:25:55.945Z",
        "success-count": 3412
      },
      "uuid": "37d3cbe5-989b-4d5f-9e03-fd5dfe8e3fb2"
    }
  ],
  "external-identifiers": {
    "pod-name": "/"
  },
  "health": {
    "bpf": "OK",
    "connected": true,
    "overallHealth": "OK",
    "policy": "OK"
  },
  "identity": {
    "id": 8,
    "labels": [
      "reserved:ingress"
    ]
  },
  "labels": {
    "derived": [
      "reserved:ingress"
    ],
    "realized": {},
    "security-relevant": [
      "reserved:ingress"
    ]
  },
  "log": [
    {
      "code": "OK",
      "message": "Successfully regenerated endpoint program (Reason: policy rules added)",
      "state": "ready",
      "timestamp": "2023-12-30T09:09:50Z"
    }
  ],
  "networking": {
    "addressing": [
      {
        "ipv4": "<IP_ADDRESS>"
      }
    ]
  },
  "policy": {
    "proxy-statistics": [],
    "realized": {
      "allowed-egress-identities": null,
      "allowed-ingress-identities": null,
      "build": 81,
      "denied-egress-identities": null,
      "denied-ingress-identities": null,
      "id": 8,
      "l4": {
        "egress": [
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":8,\"protocol\":\"ICMP\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}"
          },
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":128,\"protocol\":\"ICMPV6\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}"
          }
        ],
        "ingress": [
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":8,\"protocol\":\"ICMP\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}",
            "rules-by-selector": {
              "&LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}": [
                [
                  "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                  "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                  "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
                ]
              ]
            }
          },
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":128,\"protocol\":\"ICMPV6\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}",
            "rules-by-selector": {
              "&LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}": [
                [
                  "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                  "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                  "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
                ]
              ]
            }
          }
        ]
      },
      "policy-enabled": "audit-both",
      "policy-revision": 81
    },
    "spec": {
      "allowed-egress-identities": null,
      "allowed-ingress-identities": null,
      "build": 81,
      "denied-egress-identities": null,
      "denied-ingress-identities": null,
      "id": 8,
      "l4": {
        "egress": [
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":8,\"protocol\":\"ICMP\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}"
          },
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":128,\"protocol\":\"ICMPV6\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}"
          }
        ],
        "ingress": [
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":8,\"protocol\":\"ICMP\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}",
            "rules-by-selector": {
              "&LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}": [
                [
                  "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                  "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                  "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
                ]
              ]
            }
          },
          {
            "derived-from-rules": [
              [
                "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
              ]
            ],
            "rule": "{\"port\":128,\"protocol\":\"ICMPV6\",\"l7-rules\":[{\"\\u0026LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}\":null}]}",
            "rules-by-selector": {
              "&LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{},}": [
                [
                  "k8s:io.cilium.k8s.policy.derived-from=CiliumClusterwideNetworkPolicy",
                  "k8s:io.cilium.k8s.policy.name=allow-icmp-echo-requests",
                  "k8s:io.cilium.k8s.policy.uid=0862cc33-463e-4ce2-a0bb-8fedef061b97"
                ]
              ]
            }
          }
        ]
      },
      "policy-enabled": "audit-both",
      "policy-revision": 81
    }
  },
  "realized": {
    "label-configuration": {},
    "options": {
      "ConntrackAccounting": "Enabled",
      "ConntrackLocal": "Disabled",
      "Debug": "Disabled",
      "DebugLB": "Disabled",
      "DebugPolicy": "Disabled",
      "DropNotification": "Enabled",
      "MonitorAggregationLevel": "Medium",
      "PolicyAuditMode": "Enabled",
      "PolicyVerdictNotification": "Enabled",
      "SourceIPVerification": "Enabled",
      "TraceNotification": "Enabled"
    }
  },
  "state": "ready"
}
}

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I'm having similar issue.
Upgrade from 1.14.4 to 1.14.5 breaks ingress despite audit mode being enabled.
I'm running cilium on single node test cluster.
OS: gentoo
Kernel: 6.1.69-gentoo-dist
kubernetes 1.28.2
my helm values
cluster:
  name: kubernetes
kubeProxyReplacement: true
k8sServiceHost: <IP_ADDRESS>
k8sServicePort: 6443
operator:
  replicas: 1
serviceAccounts:
  cilium:
    name: cilium
  operator:
    name: cilium-operator
tunnel: vxlan
encryption:
  enabled: true
  type: wireguard
hubble:
  peerService:
    clusterDomain: REMOVED
  relay:
    enabled: true
  ui:
    enabled: true
    ingress:
      enabled: true
      annotations:
        cert-manager.io/cluster-issuer: "letsencrypt-prod"
      className: "cilium"
      hosts:
        - REMOVED
      labels: {}
      tls:
       - secretName: REMOVED
         hosts:
           - REMOVED
ingressController:
  enabled: true
  default: true
  loadbalancerMode: shared
ipam:
  mode: "cluster-pool"
  ciliumNodeUpdateRate: "15s"
  operator:
    clusterPoolIPv4PodCIDRList: ["<IP_ADDRESS>/16"]
    clusterPoolIPv4MaskSize: 24
ipv4:
  enabled: true
ipv6:
  enabled: false
hostFirewall:
extraConfig:
hostServices:
  enabled: true
externalIPs:
  enabled: true
nodePort:
  enabled: true
hostPort:
  enabled: true

issue is also present on cilium 1.15.0-rc.0

I’m surprised people haven’t been experiencing this before, I’ve been seing this for a while now: #28254.

Hi, came upon this issue my self.
My setup is
Following CCNP default-deny
spec:
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: UDP
          rules:
            dns:
              - matchPattern: '*'
  endpointSelector:
    matchExpressions:
      - key: io.kubernetes.pod.namespace
        operator: NotIn
        values:
          - kube-system
          -

And then I have a haproxy-ingress and a gateway httproute
CNP for ingress:
endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: <ingress-namespace>

CNP for gateway (shared):
  endpointSelector: {}
  ingress:
    - fromEntities:
        - ingress

For <= 1.14.4 all is OK
For 1.14.5, gateway/envoy is failing with 403 Access Denied, ingress still OK.

@jrajahalme -- could you take a look at this? It seems like the backport of https://github.com/cilium/cilium/pull/28126 may have caused a regression.

This is also the case on v1.15.0. (my yesterday attempt to upgrade lead to a disaster and an all-out outage)
I even tried to disable host firewall and unset policy changes (i.e use helm default), the issue still persisted.
Also, cilium-envoy was flooded with this message:
cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success

Same issue on v1.15.0 and I also see cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success in the envoy logs

Issue exists on 1.14.7 and 1.15.1

Longer version of log generated while trying to curl ingress service.
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory" subsys=envoy-filter threadID=67
level=warning msg="[cilium.bpf_metadata (egress): No policy found for <IP_ADDRESS>" subsys=envoy-config threadID=67
level=warning msg="[[C81] cilium.network: Cilium Socket Option not found" subsys=envoy-filter threadID=67
level=info msg="Envoy: Accepted access log connection" subsys=envoy-manager
level=info msg="Envoy: Closing access log connection" subsys=envoy-manager
level=warning msg="[cilium.l7policy: Cilium Socket Option not found" subsys=envoy-filter threadID=67

cilium monitor -t policy-verdict doesn't show any denied connections.
curl output
* IPv6: (none)
* IPv4: <IP_ADDRESS>
*   Trying <IP_ADDRESS>:443...
* Connected to grafana.k8s3.REMOVED (<IP_ADDRESS>) port 443
* ALPN: curl offers http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [6 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3989 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=grafana.k8s3.REMOVED
*  start date: Jan  6 15:43:55 2024 GMT
*  expire date: Apr  5 15:43:54 2024 GMT
*  subjectAltName: host "grafana.k8s3.REMOVED" matched cert's "grafana.k8s3.REMOVED"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
} [5 bytes data]
> GET / HTTP/1.1
> Host: grafana.k8s3.REMOVED
> User-Agent: curl/8.5.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/1.1 500 Internal Server Error
< date: Mon, 19 Feb 2024 14:58:07 GMT
< server: envoy
< content-length: 0
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host grafana.k8s3.REMOVED left intact

These config options don't seem to matter because disabling them doesn't change the outcome.
hostFirewall:
  enabled: true
extraConfig:
  allow-localhost: policy

Could this issue be related to this commit? d59aadc1def642694b461ae51fcfe3ce29ca6bc9

Okay, for some of these cases, I think that we may be having a similar issue to #28254, where traffic from the recently introduced ingress identity to the Service IPs is not being allowed, because the default deny is catching it.
When I replicated this scenario locally in a kind cluster, this CCNP resolved the issue:
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-ingress-to-cluster
  namespace: default
spec:
  endpointSelector:
    matchExpressions:
    - key: reserved:ingress
      operator: Exists
  egress:
  - toEntities:
    - cluster

Could I ask that the folks affected here try this and see if it fixes their issue?
However, I'm not sure about the interaction between policyEnforcementMode: always and policyAuditMode=true,, so I'm not sure if this will also fix the initial cause of this issue. But hopefully this should help us get closer if we can see what happens.
Thanks!

Okay, for some of these cases, I think that we may be having a similar issue to #28254, where traffic from the recently introduced ingress identity to the Service IPs is not being allowed, because the default deny is catching it.
When I replicated this scenario locally in a kind cluster, this CCNP resolved the issue:
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-ingress-to-cluster
  namespace: default
spec:
  endpointSelector:
    matchExpressions:
    - key: reserved:ingress
      operator: Exists
  egress:
  - toEntities:
    - cluster

Could I ask that the folks affected here try this and see if it fixes their issue?
However, I'm not sure about the interaction between policyEnforcementMode: always and policyAuditMode=true,, so I'm not sure if this will also fix the initial cause of this issue. But hopefully this should help us get closer if we can see what happens.
Thanks!
Edit: Although we are allowing to cluster in this policy, and not world, even though the Hubble flow above says world, the L7 filtering actually chooses a backend first, then makes a policy decision. But that isn't reflected correctly in Hubble, which still thinks that the flow is going to the VIP, which will be a non-cluster address. So this is why the above Policy can work.

Hmm, my inital testing (on cilium 1.15.1) does'nt seem to work. I cannot see any drops in hubble. I will continue testing.
/hw

I've just tried to update my cluster to 1.15.2 without success.
using this ccnp doesn't help.
Ingress doesn't work, there are no drops in hubble.
I have this in my logs
-> Request http from 0 ([reserved:ingress]) to 0 ([reserved:world]), identity 8->2, verdict Denied GET https://app.REMOVED/ => 0
-> Response http to 0 ([reserved:ingress]) from 0 ([reserved:world]), identity 2->8, verdict Forwarded GET https://app.REMOVED/ => 403

@adippl, as I said above, this is because the ingress identity is now used as a source identity when the flows get to Envoy, so if there is no network policy to allow the ingress identity to get to the endpoint, then traffic will be 403d.
The lines you've posted look like drops in Hubble to me. On 1.15.2, the first flow is being Denied, and then the second flow shows the 403 occcuring. Note that the identity is now world, which as far as Cilium is concerned means "outside the cluster's network", so without something to tell Cilium more about those addresses, the deny-all policy will block this traffic.

Hi, at least for my situation, I needed to exclude ingress identity from default-deny.
endpointSelector:
  matchExpressions:
    - key: io.kubernetes.pod.namespace
       operator: NotIn
       values:
         - kube-system
         - ...
    - key: ingress
       operator: DoesNotExist

@hwaastad Can you share your default-deny ccnp/cnp?

@rauanmayemir probably bad policy to update history, but anyways, added som more info to my previous example.

Folks, it seems like we've got this relatively nailed down as being mainly centred on the new ingress identity. Are there any other changes to the docs you'd like to see? If so, what are they? If not, we may be able to close this one out?

I'm planning a [potentially] painful upgrade in a week or two, will check if this resolves my previous issue.

Just trying out cilium on eks 1.29 with policyEnforcementMode=always and I've encountered the same issue
May  3 21:01:52.883: <IP_ADDRESS>:20148 (ingress) -> kube-system/cilium-ingress:31849 (world) http-request DROPPED (HTTP/1.1 GET http://xxx.us-west-2.elb.amazonaws.com/subgraphs/)
May  3 21:01:52.883: <IP_ADDRESS>:20148 (ingress) <- kube-system/cilium-ingress:31849 (world) http-response FORWARDED (HTTP/1.1 403 0ms (GET http://xxx.us-west-2.elb.amazonaws.com/subgraphs/))

These are the cluster policies I have in place
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: kube-system
  namespace: kube-system
spec:
  description: Allow kube-system endpoints to connect anywhere and receive
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - {}
    - fromEntities:
        - cluster
    - fromEntities:
        - world
  egress:
    - toEntities:
        - cluster
    - toEntities:
        - world
---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: kube-dns-cluster-ingress
spec:
  description: Policy for ingress allow to kube-dns from all Cilium managed endpoints in the cluster
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: kube-system
      k8s-app: kube-dns
  ingress:
  - fromEndpoints:
    - {}
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: kube-dns-cluster-egress
spec:
  description: Policy for egress allow to kube-dns from all Cilium managed endpoints in the cluster
  endpointSelector: {}
  egress:
  - toEndpoints:
      - matchLabels:
          io.kubernetes.pod.namespace: kube-system
          k8s-app: kube-dns
    toPorts:
      - ports:
          - port: "53"
            protocol: UDP
        rules:
          dns:
            - matchPattern: "*"
---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: cilium-health-checks-cluster-ingress
spec:
  endpointSelector:
    matchLabels:
      'reserved:health': ''
  ingress:
    - fromEntities:
      - remote-node
  egress:
    - toEntities:
      - remote-node
---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-ingress-to-cluster
spec:
  endpointSelector:
    matchExpressions:
      - key: reserved:ingress
        operator: Exists
  egress:
    - toEntities:
      - cluster

the last one being the one from this issue. Also the app that has the ingress that's trying to be reached has
        - fromEntities:
            - world
            - ingress

In this case it seems that the ingress entity is trying to connect to the kube-system/cilium-ingress endpoint (that I don't see with kubectl get cep)

Fixed it by changing the suggested ingress policy to
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-ingress-egress
spec:
  endpointSelector:
    matchExpressions:
    - key: reserved:ingress
      operator: Exists
  ingress:
    - fromEntities:
        - world
  egress:
    - toEntities:
      - cluster

is this the right thing to do?
I think in the doc would be nice to show a somehow "hardened" setup with a deny all by default, that allows core components like dns/hubble/kubernetes-api/ingresses to be functioning and shows an example on how to expose a service in another namespace to the world with an ingress

Thanks for that @0xalex88, I agree that showing a hardened setup is a good idea, #31436 is open already, and I pinged @sayboras just now to take another look at it.
I should note though that you can't have an Ingress and the backend in separate namespaces. Doing that requires Gateway API (the API for cross-namespace references needs to be carefully design to prevent further security problems, which we did for Gateway API.)

I'm happy to report that the upgrade went very smooth, no more 403 issues with a CCNP rule for reserved:ingress. 😭

Thanks for that report @rauanmayemir, that's awesome. Everyone else who's responded on here, if you can check in, we'd appreciate it.
I think it seems likely that adding the rule for reserved:ingress is the way to resolve these issues, so we would like to close this at some point. But I'd also like to hear if this is actually the case.

I faced this issue when I tried to harden my cluster with policyEnforcementMode: always.
Thank you @0xalex88 for the CCNP examples!

After applying the policy to match the reserved:ingress we can now reach the backend, but for about only 5%-10% of the requests. Most 90% of requests fail with this error:
HTTP/1.1 503 Service Unavailable
content-length: 91
content-type: text/plain
date: Tue, 04 Jun 2024 10:51:36 GMT
server: envoy

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Not sure what's missing there because I don't see any connections dropped by cilium (checked with hubble...)

Hey everyone on here - I've recently merged #33698, which added a bunch of extra information about the reserved:ingress identity, and moved the existing example of Network Policy to make it more visible. The new version is available at https://docs.cilium.io/en/latest/network/servicemesh/ingress/ for now (note the latest version). As of the 1.16 release, due real soon, this will be promoted to the `stable docs.
If you find this issue because of a similar problem, please review these docs and the examples and see if that helps. The example also includes a more up-to-date CiliumNetworkPolicy for handling exempting Ingress traffic from a broader deny-all.
I believe that this documentation should enable folks who find this issue to solve things themselves, so I'll close this one. Please comment if you would really like it reopened, but I think that this issue has got a bit large.
In particular, @project-administrator, if your problem still persists after you've reviewed those docs, please open another issue.


perf(transformer): logical assignment operator transform use SparseStack

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#5960  👈
#5959
#5942
#5940
#5945
#5941
#5944
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite


WebUI enhancements; multilanguage

Enhancements in the WebUI (some nicer controls)
Uses LoxBerry's notifys instead of own warnings
Support for Multilanguage (see templates folder)
Saves and restore config when upgrading plugin
Restarts server after upgrading

Looks good to me, merging!


Pagination for classrooms and assignments

We still need to add pagination for the classrooms list.

Hello @srallen ,
I am interested to contribute. Please provide more info. Thank you


fix: handle missing angular dependencis error

Follow up to #87, our observability picked up on another unhandled internal error:

https://ui.honeycomb.io/gatsbyinc/environments/buildbot-production/datasets/-netlify-build/result/ytrSVXhBnk3/trace/bah7dJwdaJn?fields[]=s_name&fields[]=s_serviceName&span=56f21e0e4aaa2eb1

This time is the unhandled resolve exception. This should take care of that hopefully 👍

it looks like the test runs are installing the plugin from NPM, as opposed to using the local one 🤔 https://github.com/netlify/angular-runtime/actions/runs/7004802463/job/19053297061?pr=91#step:6:285

@ascorbic  @Skn0tt done. Mind re-👀-ing?


Compress with gzip on npm run start

Is your proposal related to a problem?
It's annoying having to deploy and check whether your improvements have worked or not, we can't get the localhost server to verify if lighthouse in devtools passes to a reasonable score.
Describe the solution you'd like
Whilst developing and fixing issues with lighthouse in devtools, it would be nice, on localhost we can provide an argument npm run start --compress=true so locahost serves compress gzipped CSS and JS etc.
This will prevent me from finding workarounds and increase my productivity tremendously.
Alternatively a flag on the package.json can also be an option, but I prefer the cli option, as it can be optionally added on cli npm start --compressed this way I can use it on the fly only when I need to and not always active/inactive.

I don't think you should ever do any performance testing while you're running the application in development mode.
What I'd suggest is that you create a build of your application first, serve it with something which supports gzip compression (i.e. serve-gzipped) and then test the performance.
If you just run tests while in development mode, the tests won't be authoritative.


Current pose changes too rarely and no rviz topics other than current_pose_estimate

I managed to get it working with Orin Nano, zed2i and odometry from it. The main problem is that current_pose_estimate changes too rarely to be usable althoug the topic hz is around 10. Is it a parameter issue? How can I improve the frequency?
The other issue is more like a question, what is the "normal scene" of topics and rviz topics when cslam is correctly working?
Also, attached a video with the example of current_pose_estimate update rate in rviz.
https://github.com/MISTLab/Swarm-SLAM/assets/94308727/7c92892b-d256-476b-b313-a5a81316553c

Hi! Thanks for your interest in our work :)
The delay you are experiencing is strange to me.


fix: guard against unnecessary deselect_all event fired

@vakata done! let me know if you need me to do other changes.

Thank you.
I am terribly sorry, I believe I missed something in the explanation on why I can't merge this - you are checking against the state selected property, while the check should be against the current tree state.
The idea is that deselect_all should only be called if there is something currently selected, so that the newly selected items can be applied. For example:

You have a tree with node A selected
You pass in a state with no nodes selected (selected is an empty array)
The expected result is that no nodes should be selected after applying the state

That means that the above PR will fail, as it will not deselect the A node.
Best regards,
Ivan


Scripts don't appear in Tools-->Customize anymore

Hi Toni,
sorry, I didn't find another way to contact you. There is no problem with your scripts but  since yesterday I have the problem that the scripts in %ProgramData%\CODESYS\Script Commands aren't recognized anymore by CodeSys. Do you have any idea what my problem is?
Thanks for your answer, and sorry for making an issue....
Thomas

Hi, did you maybe upgrade to the latest codesys sp19? Maybe they changed the scripts folder. I still run sp18. I will upgrade to sp19 when I have the time and check it out.

Thanks for your answer. I had 18.2 and now upgraded to 18.5. After the
upgrade it worked again.
Toni Kucic @.***> schrieb am Mo., 7. Aug. 2023, 21:19:

Hi, did you maybe upgrade to the latest codesys sp19? Maybe they changed
the scripts folder. I still run sp18. I will upgrade to sp19 when I have
the time and check it out.
—
Reply to this email directly, view it on GitHub
https://github.com/tkucic/codesys_workflow_automation/issues/3#issuecomment-1668447367,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJOXJZ3ZULLEGEJYUMCUEVDXUE5TPANCNFSM6AAAAAA3BZBGDQ
.
You are receiving this because you authored the thread.Message ID:
@.***>


Heuristic compilation database

The need to create a compilation database to use clangd at all is a large hurdle.
For projects with complex builds we're probably stuck with this - running clang requires accurate flags. For projects with simple builds, we can fall back to guessing those flags. The MS VSCode plugin has some success with this approach.
Our current fallback is a simple function of the filename, which is pretty limited: we can't reasonably do a ton of IO in getFallbackCompileCommand(), and it can never support background indexing etc.
Creating a fallback CDB addresses these problems: it lets us amortize the discovery across files, gives us somewhere to put discovered state, and we can scan for files to index.
This should probably have several passes:

first we do the "regular" scan up from the source for compilation databases. We don't want to create a heuristic one if there's a real one further up.
then if we didn't find anything, we try to detect a project root for the CDB. This should be fairly cheap (e.g. check for a few artifacts like makefiles) as we're looking at a bunch of candidates
if we select one, do detailed scanning (determine include paths etc). This can do more work (e.g. recursive directory walking) but not an unbounded amount.
when a particular file is queried, we can do a bit of additional work - we can use the filename and afford to read the content

This is going to make invalidation more complicated. We don't do invalidation today, so this is "just" technical debt.

Detection ideas:

version control root: .git etc
build systems we can't parse: Makefile, BUILD, CMakeLists (here but not in parent dir, for the root)
misc files that happen to live in project root a lot: configure.in, LICENSE.txt, package.json...
ad-hoc detection of particular projects: kernel, UE4, stdlib, who knows

Misc stuff:

do we always want background-indexing on? for all files?
do we want all different types of detected projects to use a single set of heuristics with options, or should we split off e.g. UE4CDB and find ways to "mix in" the desired heuristic behaviors?
how much of this should we try to push into libtooling for other tools?

Per-file ideas:

when the CDB is queried for a file, we can read its #include s and refine our guesses about where the -I lines should be rooted. We shouldn't solely rely on this though, cross-file information will make include-completion work.

Gonna throw into the mix that when you find a file, asking the client to provide the flags would be super great for YCM, as we can query the user's ycm_extra_conf file and for each discovered file, we can give you the correct flags (or None, if it's not a self-contained TU)

asking the client to provide the flags

Does this need to work for backrgound index? I'm worried about scale.
The critical case is when we open a big project that's already indexed. Before the index can be used, we need to a) create the CDB, b) list all the files, c) load all the shards and check if they're up to date (which depends on the compile command). Whether we do this during A or C, that's a lot of LSP traffic and editor-side processing can this be fast enough?
(workspace/configuration is likely the LSP vehicle for this)

Detection ideas:

Before we get to all the heuristic stuff mentioned, what about InitializeParams.rootUri?

* ad-hoc detection of particular projects: linux, UE4, stdlib, who knows

Wouldn't these fall more into the "For projects with complex builds we're probably stuck with [having them provide a CDB] - running clang requires accurate flags." case? Perhaps we could work with these projects to have their build systems include a "generate CDB" target?

maybe we can textually parse some useful bits from CMakeCache, build.ninja etc even if we can't understand them fully. Similarly ycm_extra_conf?!

If it's a CMake project, perhaps there's a way to "re-run CMake with the same arguments as last time", and we can just add the -DCMAKE_EXPORT_COMPILE_COMMANDS=1?

Does this need to work for backrgound index?

yes ideally, as the user experience we all want is for the full index to just work. The use case here is where is have an arbitrarily complex build system where I write some code (ycm_extra_conf) to query that build system for the right flags for a given TU. YCM has (had?) 1000s of users doing this sort of thing, and the number one request over the years is "why doesn't Go To Definition work". Of course, it does work now that we use clangd but for users who can't make a compilation database, we'd love to offer the same or similar level of functionality.

The critical case is when we open a big project that's already indexed. Before the index can be used, we need to a) create the CDB, b) list all the files, c) load all the shards and check if they're up to date (which depends on the compile command). Whether we do this during A or C, that's a lot of LSP traffic and editor-side processing can this be fast enough?

This is a good point. In the case of YCM (which is really ycmd) it's already a server process so this work would not be blocking the user. Perhaps in this mode, the requests can be batched or batched and throttled, but it's certainly worth thinking about.

Before we get to all the heuristic stuff mentioned, what about InitializeParams.rootUri?

I'm not sure this tells us anything meaningful about the codebase in practice. For cd /tmp; $EDITOR ~/test.cc:

vscode says null
coc.nvim says "/tmp"
ycm says $HOME

So maybe it tells you something about the editor! But similarly I'd guess coc is probably sensitive to your initial working directory, ycm to the first file you open, and vscode to the way you opened the window. I don't think it's reasonable for something so important to depend on this phase-of-the-moon stuff.
If the editor is sure it knows best, it can send the compile-commands directory, and we should respect that even if there's no "real" CDB and we need to use heuristics.

If it's a CMake project, perhaps there's a way to "re-run CMake with the same arguments as last time", and we can just add the -DCMAKE_EXPORT_COMPILE_COMMANDS=1?

Yeah, for better or worse, CMake keeps all your settings from last time by default so that's DCMAKE_EXPORT_COMPILE_COMMANDS=1. It seems a bit rude to do this without asking first maybe, but we can prompt.

Wouldn't these fall more into the "For projects with complex builds we're probably stuck with [having them provide a CDB]

Maybe! I meant more "there will be limits" than "we shouldn't try" though.
There's a reasonable concern that getting it half-working rather than obviously-broken will muddy the waters and discourage people from fixing their configs, leading to bad outcomes in the end. so we should be wary of UX and keep the user informed. We also shouldn't let the perfect be the enemy of the good in some of these cases though...

it's already a server process so this work would not be blocking the user

It's not blocking keystrokes, but if it takes a minute instead of a few seconds to load the index then it might be blocking the user!
(We could separate out the validating the shards and put it on a different queue if it becomes a lot slower than loading them... everything is solvable with enough code)

I'm not sure this tells us anything meaningful about the codebase in practice. For cd /tmp; $EDITOR ~/test.cc:

ycm says $HOME

I think that's a fairly unusual case. For 1000s of projects in the world with .ycm_extra_conf files, YCM would return the path of that file, which is most likely the project root.

It's not blocking keystrokes, but if it takes a minute instead of a few seconds to load the index then it might be blocking the user!

Blocking only if loading the background index is blocking real-time operations, which is isn't in the same way that building the background index isn't blocking real-time operations in practice. I do take your point that for a large project, where the index is fully up-to-date, then the querying for the flags may dominate load time, but in any scenario where we have to parse the resulting TU, the expectation is that parsing time should dominate rather than dtermining-flags-time.
I do feel that the flexibility of providing arbitrary flags with arbitrary complexity for the very real projects where generating a compilation DB Is not something most users can do is worth the potential extra work.
To make this concrete, right now, in order to have a full index I have to run bear make COMPILER=clang and wait for the whole project to build (like 10 minutes) so that I have a compilation database. That's a lot more time than I would expect this process to take.
I wonder if there's a middle-ground here. Like, could clangd even record the compilation command used in the index. Then loading the index can be just-as-instant, and querying to see which TUs' compilation commands has changed could be done on a slow tick background process.
Just thinking out loud.

I think you are making this so big.
take root directory of workspace. any .h/.hpp file is a header, any .cpp/.cc file is source.
run c++/g++/... command to find installed compiler and use this as driver to query system headers. (--query-driver flag)


📝Update Python Readme (Fix #172 ) And Added new Python Solution eval_py

Let me know what you think about this guys.

Please add the entire question on top as a comment. Check the other codes in the repository for reference.

Check it out now!

Add the full question from Link. Check the other codes in the repository for reference.

I have added objective, task, input, output as specified in the problem.


Fix #1330 Support capital letters in mimetype in is_json

Your bugreport is fine, but I think this should be fixed on a broader scale, i.e. make response.mimetype always lowercase.


Missing images during file conversion

This issue is similar to #294 but it looks like progress on that has stalled and my use case is slightly different as I am using the rebase_relative_paths extension.  I've tried a number of different combinations of  changing the working directory and providing a resource path but no matter what I can't get pypandoc to convert a document with images.  Maybe I'm missing something but  the document converts fine using an almost identical command provided to the pandoc binary.  I have tried multiple versions of pandoc, the version listed below was just the last one I tested against.
I've created a repo with an example that will hopefully help.
Using pypandoc

>./main.py
[WARNING] Could not fetch resource documents/doc1/doc/../resources/image.png: PandocResourceNotFound "documents/doc1/doc/../resources/image.png"

Running the same command in pandoc doesn't produce any warnings about missing resources.

> pandoc documents/doc1/doc/test.md --from=markdown+rebase_relative_Paths --to=docx -o output/result.docx
>

> pip show pypandoc
Name: pypandoc
Version: 1.10
Summary: Thin wrapper for pandoc.
Home-page: https://github.com/JessicaTegner/pypandoc
Author: b'Juho Veps\xc3\xa4l\xc3\xa4inen'
Author-email<EMAIL_ADDRESS>License: MIT
Location: /Users/username/.pyenv/versions/3.10.1/envs/pypandoc-test/lib/python3.10/site-packages
Requires:
Required-by:

> python --version
Python 3.10.1

> pandoc --version
pandoc 2.18
Compiled with pandoc-types 1.22.2, texmath 0.12.5, skylighting 0.12.3,
citeproc 0.7, ipynb 0.2, hslua 2.2.0
Scripting engine: Lua 5.4
User data directory: /Users/kreby/.local/share/pandoc
Copyright (C) 2006-2022 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

Having the same issue. Happens even when using --resource-path to the directory, at least for DOCX conversions.

Had the same issue and setting sandbox=False when using pypandoc fixed it for me, but see my full comment here

I'm awear of thise, and discussion will continue in #316


Remove reference to AzureRm

There is still a reference to Import-Module AzureRM when it is not needed when using Az under the following:
https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logs#turn-on-logging-using-powershell

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: aa60fa41-f753-9c55-c799-a870bdb219a4
Version Independent ID: 786b27ec-7f4d-ad35-e26d-5832c072536e
Content: Diagnostic logging in Azure Databricks - Azure Databricks - Workspace
Content Source: databricks/administration-guide/account-settings/azure-diagnostic-logs.md
Service: azure-databricks
GitHub Login: @mssaperla
Microsoft Alias: saperla

@byronbayer - Thank you for sharing feedback. We are taking look in to this will get back to you soon

@leifbro - Could you please validate and update as appropriate?

Thanks for your dedication to our documentation. Unfortunately, we have been unable to review your issue and apologize for the delayed response. The requested updates have not yet been made. The timeline for resolution varies based on resourcing. We've created an internal work item (DOC-6994) to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary.
#please-close

#please-close


Installation on a fresh debian 9 results in errors

It seems like that your script writes the database password in some source code? In my test setup I entered 12345 as password for the database.
Unhandled exception: Expected String, not 12345 at line 7, column 13 (YAML::ParseException)
  from /usr/share/crystal/src/yaml/nodes/nodes.cr:30:9 in 'raise'
  from /usr/share/crystal/src/yaml/from_yaml.cr:0:7 in 'new'
  from /usr/share/crystal/src/yaml/from_yaml.cr:165:3 in '???'
  from /usr/share/crystal/src/string/builder.cr:0:5 in '???'
  from src/invidious/videos.cr:0:1 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:47:14 in 'main'
  from /usr/share/crystal/src/string/builder.cr:11:3 in 'dump'
  from _start
  from ???

Hi, it writes to the config.yml file.
https://github.com/tmiland/Invidious-Updater/blob/f0ecc8b5cd3b4321b086c8bb9c6abd9a3a89effd/invidious_update.sh#L169
What version did you use? Pushed a new update last night. (version 1.1.2)
I run the script on a fresh Debian 9, got no issues as of 1.1.2

I can confirm this issue. Must be a bug with Invidious not able to handle "12345" as password.


Improve Vector2 / 3 / 4 normalized() classref.

Mention that the results will be unreliable with initial vector near zero.
Helps address #74852
Notes

The alternative is to add an epsilon to normalized(), and e.g. return zero in these cases and note this in classref.

Doing the above would technically be borderline compat breaking, although we may get away with it (you could argue that values that low were giving spurious results anyway). normalized() isn't an ideal function to be using on input that could be near zero, because of the difficulty involved in detecting failure ( see https://github.com/godotengine/godot/issues/74852#issuecomment-1465709454 )
We probably just have to pick our poison with this, we can do another PR with an epsilon for normalized().

Needs a rebase after another PR made some changes to these descriptions. Also, what about Vector4?

Needs a rebase after another PR made some changes to these descriptions. Also, what about Vector4?

Rebased and also added Vector4.

Thanks!


Replace Ctx.CustomObjects type with thread safe and secure ObjectMap API

Hey! 👋
I stumbled across this project a while ago and currently, I'm thinking about using it in the upstanding rework of yuri, so I've decided to try to give some improvements to this package, if wanted. ^^
First of all, using a raw map[string]interface{} as Ctx.CustomObjects might bring some nasty issues. Concurrently reading on maps in go is fine until you read/write on maps in parallel, which will lead to a panic.
Here you can read more about this with some handy examples:
https://qvault.io/2020/03/19/golang-mutexes-what-is-rwmutex-for
So i've simply wrapped the map with a sync.RWMutex which blocks access to the map while writing to it and also providing some nice API endpoints for accessing the CustomObjects field.
Also, I've simplified the inline object creation in the embed arrays of helpCommand.go.
Additionally, I've put the example from the README into the example directory so that people can test around with the package using the provided example.

Okay, I've changed m to innerMap and mx to mutex for more clarity.
Also, I've updated the version of timedmap which got also an update this morning. ;)

Alright  seems good. Thank you!


Update README.md

add lazy

Dead project?

Hey, thanks for the PR. If org some sort of standard? If not, I'd rather not change it, and if we're to change it, can you also make the change in the other places in the README.md as well?


Remove movie api key from workflows

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 54.01%. Comparing base (838d0bb) to head (09a1fdc).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #384   +/-   ##
=======================================
  Coverage   54.01%   54.01%
=======================================
  Files         168      168
  Lines        3710     3710
=======================================
  Hits         2004     2004
  Misses       1706     1706

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


fatal error: 'tr1/functional' file not found

pficommon does not build on OS X 10.9 or later, because it uses the transitional "tr1" namespace which is no longer available in OS X 10.9 or later:
pficommon-26b6655092adc95649c39706b37502e466e5efe9/src/concurrent/../lang/function.h:35:10: fatal error: 'tr1/functional' file not found
#include <tr1/functional>
         ^
1 error generated.

Note: if you want to retain compatibility with OS X 10.8 or earlier, you have to continue using "tr1" on those systems. So you'll have to detect whether "tr1" exists or not and act accordingly.

Thank you for reporting. We made pficommon for OS X 10.9+ https://github.com/gwtnb/homebrew-pficommon/ but porting some functions to OS X 10.9+ takes time. We are using pficommon primarily for GNU/Linux, so we will port to OS X 10.9+ if we have to do so.

I'm writing on behalf of the MacPorts project, where we currently have pficommon 83ea9e102fbe9dfd788f49b69f18585355f64692 from 20141020. It only builds on OS X 10.7 and 10.8, but not on earlier or later versions of OS X; this is somewhat limiting.
It looks like in the Homebrew formula you referred me to, an earlier version of pficommon from 20140813 is being used. I suppose we can downgrade pficommon in MacPorts to an older version if that will bring compatibility with a wider range of OS X versions, but it would be good to get the current version working on more OS X versions. The problem I mentioned above, about the tr1 namespace, isn't exactly an OS X-specific issue; I imagine you would see the same problem on any other system that has moved from the transitional implementation to the final implementation.

Your latest code still does not build with libc++.
You don't need a Mac to reproduce this issue, since libc++ is available for many systems.


Lunr index building is killed with OOM error

Original report is here https://github.com/olivernn/lunr.js/issues/305#issuecomment-336872069
Link to repro repo: https://github.com/emirotin/lunr-failure.
You need to untar theJSON dump file before running the code.
The process is using around 1.5GB RAM when it crashes.
I'm on MBP 2017 with 16GB RAM and the overall memory pressure when running the tests is pretty low (App Memory combined is about 8GB, the chart is all green).

I get similar results, it gets to about 54K entries and then massively slows down, I got bored waiting for it to crash...
I re-ran the command again, but this time increased the amount of space available to node to 4GB:
$node --max-old-space-size=4096 build-index.js
It got to 156K entries before slowing down again, so I think you just need to increase the amount of memory you give node.
My guess is it slows down as the inverted index gets large and it starts to run out of available heap space because v8 is having to do a lot of heavy lifting making space for the inverted index, probably its memory space is getting more and more fragmented.
If your data contains multiple languages my guess is that using the appropriate stemmers and stop words is significantly reducing the amount of data that needs to enter the index, hence it manages to get further before node runs out of space.
Out of interest, what is your use case for this index? I'm interested to know what kind of query performance you manage to get, if it ever manages to complete building the index... At this scale you might be pushing the limits of what Lunr is capable of, certainly the index that it might eventually generate is going to be very large, probably too large to send to a browser?

Thanks for your thoughtful investigation!
It's quite possible indeed that I've hit the limits of Lunr and have to use another solution, though Lunr is really easy to work with and I had good experience using it before (with much smaller amount of doc though), so it was my first candidate.
Here's my use-case.
There's a remote DB of docs of some sort (essentially they're questions similar to those for pub-quizzes).
The authors often use it to check if the new questions contain the facts or ideas previously used by other authors.
It has several issues:

may be slow or down sometimes
if somebody can intercept the queries they can guess the themes for the future questions
the built-in search itself is known to sometimes miss the results

The questions are mostly in Russian, with specific words in English, more seldom in Latin, French, Italian, German, etc.
So I want to create the all-local apps to runs the searches against this DB.
For that I'm (going to be)

creating the replica of this DB on my server
indexing this replica and building the search index suitable for full-text search with specific features (being able to only search by specific fields, powerful language-aware stemming, optionally but nice to have fuzzy search with acceptable word distance)
creating the local apps (first desktop, then potentially mobile) that will periodically download the search index (and maybe the SQLite DB as well) and run all the searches locally, offline, against the local index / DB. The app can download the updates in the background, so it's not that bad if it has to fetch 500M every now and then (for the desktop app, I didn't think about the mobile strategy yet).

As JS is my language of choice I'd like to have the server in Node.js, desktop app in Electron / React, and mobile apps in React Native, so some JS-first solution is really nice to have :)
The provided dump is the current state of the production data, and it's slowly growing over the time, so I need the scalability up to at least 1M docs.

Interesting, thanks for the context. So I think it might be pushing Lunr too far, I've not heard of people with such large indexes. That said, I'm now interested to see how far we can push the current implementation!
From the testing that I've done so far it looks like the bottleneck is the GC, and allocation size. I reckon getting indexing to first complete, and then not take hours is going to be an exercise in tuning v8 and its GC. I'm slightly concerned that it is slowing down so much while just adding the documents, it hasn't got to the part where it constructs the TokenSet which might also put a lot of extra pressure on the GC.
My next steps is going to launch an instance with a decent amount of memory and give it all to node. How any documents are in the dump that is part of that repo?
Even if we don't manage to get a working solution, I'm sure we'll find a bunch of optimisations that might benefit more moderate indexes.

:100:
I managed to finish the indexing today with 8GB or 10GB max memory limit for Node (and it takes about 6-7 minutes), and the final index construction from the builder takes negligible time. But then serializing the index to JSON takes too much time unfortunately. I've been able to make it work with bfj but it's been taking too much time to stream it to disk so I had to abort it. I also expect reading such big index will also take ages though I may be wrong.
ATM I'm looking into alternative approach (SQLite which I already have as the intermediary storage + FTS5 + Snowball stemmer), but I'd be happy to get Lunr working or at least help improve it. It's definitely the easiest and friendliest FTS in JS world.
The provided dump has all the 300K+ docs that exist in the DB as of now.

I noticed while adding some more logging to the script that the inverted index size was very large, e.g. >400K keys. This implies that there are over 400K unique tokens in your documents. Apparently there are ~ 200K english words in OED:

The Second Edition of the 20-volume Oxford English Dictionary contains full entries for 171,476 words in current use, and 47,156 obsolete words. To this may be added around 9,500 derivative words included as subentries.

Lets say there are the same for Russian (I don't know), that then gets us to the ~400K mark, but does you data actually contain most words in the dictionary? I don't know what kind of data it is, does that number of unique terms sound feasible?
Running locally I managed to get all the documents added to the index, but then trying to serialise the index just hangs. I suspect that JSON.stringify isn't really cutout for such large objects, I think this is what you found too based on your previous comment...

Russian dics may indeed contain up to 200K words (that's the case for the
Dahl's dictionary), but that including lots of obsolete, local, and not
often used words, and including derivatives.
My docs are written in modern natural language and are usually short (1-3
sentences).
They also contain sources which are bibliographical references and URLs.
They sometimes contain non-Russian words, but that are individual terms or
titles that shouldn't significantly contribute to the number of unique
tokens.
So overall it's very unlikely for my corpus to really contain 400K unique
tokens.
And yes wrt serialization, as I wrote in the prev comment I've found the
way to serialize it with bfj lib, but it's unfeasibly slow.
On Thu, Oct 19, 2017 at 8:09 PM Oliver Nightingale<EMAIL_ADDRESS>wrote:

I noticed while adding some more logging to the script that the inverted
index size was very large, e.g. >400K keys. This implies that there are
over 400K unique tokens in your documents. Apparently there are ~ 200K
english words in OED:
The Second Edition of the 20-volume Oxford English Dictionary contains
full entries for 171,476 words in current use, and 47,156 obsolete words.
To this may be added around 9,500 derivative words included as subentries.
https://en.oxforddictionaries.com/explore/how-many-words-are-there-in-the-english-language
Lets say there are the same for Russian (I don't know), that then gets us
to the ~400K mark, but does you data actually contain most words in the
dictionary? I don't know what kind of data it is, does that number of
unique terms sound feasible?
Running locally I managed to get all the documents added to the index, but
then trying to serialise the index just hangs. I suspect that
JSON.stringify isn't really cutout for such large objects, I think this
is what you found too based on your previous comment...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/olivernn/lunr.js/issues/306#issuecomment-337974293,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgGCJh554NykD6AwFOCFTw3HwbR4MPtks5st4JDgaJpZM4P6mt7
.

These errors (FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory) are happening to me too. Error happens for me on the ~160th time calling this.add(doc);. Each of my documents is has a large "body" (think long blog post). Is this expected?

Node.js: v10.15.3


option to always include a set of files

is there a way to include files to each compiled file?
options: {
  require: ['common/mixins.jade', 'common/functions.jade']
  data: …
}

You can add +common/mixins.jade\n+common/functions.jade\n to every file with grunt-contrib-concat.

@loominade By the way, why you create two accounts, Müller? @fuddl

one of it is private, one is professional


Check if pose pipeline can see tags and error if not

This can allow us to skip pose filtering if there's no tags in view (pose has to be wrong in this case).

Limelight makes this quite difficult from what I can tell, but its a simple change with PhotonVision

the limelight api is terribly documented, its the 7th entry in the botpose https://github.com/LimelightVision/limelightlib-wpijava/blob/main/LimelightHelpers.java#L511


PM-525 ReadTheDocs documentation

Integrates the documentation with ReadTheDocs

Uses sphinx-js to build the documentation.
The documentation is generated by ReadTheDocs.
The old documentation folder (tutorials) has been deleted and the .md and .rst files are now in docs/source

Sweet, thanks German!
I'll address some little things when I rebase another PR I have which does some docs stuff. For reference, docs can now be found here: https://gnosisjs.readthedocs.io/en/latest/api-reference.html


[BUG]: /demo page should only be accessible in development builds

Description 📝
We have a demo page that is used to test out components. Right now this is in production at https://syntaxui.com/demo.
We should remove this page from production builds.
Link 🔗
https://syntaxui.com/demo
Steps to Reproduce 🔄

Go to https://syntaxui.com/demo

Screenshots 📸
n/a
Expected Behavior 🤔
The /demo page should only exist in development and not in production
Actual Behavior 😱
The /demo page exists in production
Environment 🌍
n/a
Additional Information ℹ
No response

removed in 8b9f5c9


mark synchronized as nothrow

also added @nogc as test-case
not @safe/pure because the mutex is __gshared data

nice, thanks!


dbt_utils.deduplicate macro returns zero rows when deduplicating a CTE with NULL column (Redshift)

Describe the bug
dbt_utils.deduplicate returns zero rows when it deduplicates a CTE with a column that consists entirely of NULL values.
Steps to reproduce
Given model code as:
with data as (

    select
        '2021-04-03 23:00:26'::timestamp as ts,
        1 AS id,
        NULL AS null_field
    UNION ALL
    select
        '2021-04-03 23:00:26'::timestamp as ts,
        1 AS id,
        NULL AS null_field
),

data_deduped AS (
    {{ dbt_utils.deduplicate(
        relation='data',
        partition_by='id',
        order_by='ts desc',
    ) }}
)

select * from data_deduped

it is compiled by dbt on Redshift adapter into the following SQL:
with data as (

    select
        '2021-04-03 23:00:26'::timestamp as ts,
        1 AS id,
        NULL AS null_field
    UNION ALL
    select
        '2021-04-03 23:00:26'::timestamp as ts,
        1 AS id,
        NULL AS null_field
),

data_deduped AS (
    with row_numbered as (
        select
            _inner.*,
            row_number() over (
                partition by id
                order by ts desc
            ) as rn
        from data as _inner
    )

    select
        distinct data.*
    from data as data

    natural join row_numbered
    where row_numbered.rn = 1
)

select * from data_deduped

Expected results
         ts          | id | null_field
---------------------+----+------------
 2021-04-03 23:00:26 |  1 |
(1 row)

Actual results
 ts | id | null_field
----+----+------------
(0 rows)

Screenshots and log output

System information
The contents of your packages.yml file:
packages:
  - package: mjirv/dbt_datamocktool
    version: 0.2.0

Which database are you using dbt with?

[ ] postgres
[x] redshift
[ ] bigquery
[ ] snowflake
[ ] other (specify: ____________)

The output of dbt --version:
Core:
  - installed: 1.2.1
  - latest:    1.3.0 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - redshift: 1.2.1 - Update available!
  - postgres: 1.2.1 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

@yauhen-sobaleu thanks for opening this!
If there are two null rows and a non-null row, does it still lose the null rows? I assume so but your repro steps don't say one way or the other.

I have the same issue. I think it's the natural join that is the culprit here. It joins on all your selected column, and if you have columns where your value is null. It will not return that row.

Yes the problem is in natural join that is not null safe. What I had to do is to add another function parameter that explicitly accepts the list of columns to select from.
Using deduplicate macro from dbt-utils is quite dangerous right know. You never know when NULLs will come out and corrupt your data.
{%- macro deduplicate(relation, partition_by, order_by, columns) -%}

    WITH row_numbered AS (
        SELECT
            {% for column in columns %}
            {{ column }},
            {% endfor %}
            ROW_NUMBER() OVER (
                PARTITION BY {{ partition_by }}
                ORDER BY {{ order_by }}
            ) AS rn
        FROM {{ relation }} AS _inner
    )

    SELECT * FROM row_numbered WHERE rn = 1
{%- endmacro -%}

OK thanks for clarifying! I'd welcome a PR that resolved this (and an addition to the integration tests to ensure it stays fixed)

For the record another user hit this issue. See the slack here https://getdbt.slack.com/archives/CBSQTAPLG/p1679007195056989
and also my summary here https://getdbt.slack.com/archives/CU4MRJ7QB/p1679053101978089
TL;DR:  The switch in #548 to use a select distinct natural join not only introduced this NULL bug, but also very poor performance compared to a more sane approach and a future bug where data types that are not comparable (such as JSON in Postgres) will cause the natural join to fail.
My humble suggestion is to revert to the old method of using dbt_utils.star

I also hit the same issue and find out this issue ticket. I am using DuckDB, and the problem also comes from natural join on columns containing NULL values.

OK I've just spent an afternoon staring at the old and new versions of this macro, and no simple solution has fallen from the sky.
I would love to be able to revert to using dbt_utils.star. However, that requires a real object to be persisted in the database so that its columns can be found. That means no CTEs, in particular no ephemeral models. And that means we're breaking the backwards-compatibility commitments of utils v1.0, because this version of the macro accepts anything.

A couple alternative ideas:

Force a database-specific implementation in dbt-utils (by raising a NotImplementedException) rather than attempting a cross-database one.
Move deduplicate to dbt-core so dbt adapters are encouraged to implement it.

Unless I'm missing something, these would also require star (or in the dbt-core timeline, probably adapter.get_columns_in_relation directly) to work correctly. I guess that this would be a roll-forward situation, where the old buggy behaviour remains and people can move to a correctly-implemented alternative. Even so, I don't think this is a fit for dbt-core - it doesn't feel like it's inherently part of SQL. More like width_bucket or something where Snowflake has a one-line solution but is the outlier.
So a new macro then?
Probably! I think we need to chalk up another win for the humble null and abandon this one with a warning message inside of it. Then create a new macro like remove_duplicates() with some to-be-determined signature. It would be nice for it to continue to support arbitrary CTEs (maybe you can pass in a set of columns manually?) but it must handle nulls correctly, so the default implementation goes back to the row_number strategy.
It strikes me that dbt_utils.pivot doesn't try to be clever; it asks for the column values to be passed in. Maybe we should take a leaf from its book: dbt_utils.remove_duplicates(from, return_columns, partition_by, order_by)
For a model:
{{ dbt_utils.remove_duplicates(
  from=ref('some_model'),
  return_columns=dbt_utils.get_filtered_columns_in_relation(ref('some_model')),
  partition_by='id',
  order_by='order_date desc'
)}}

For a CTE:
{{ dbt_utils.remove_duplicates(
  from='my_cte',
  return_columns=['id', 'order_date', 'status'],
  partition_by='id',
  order_by='order_date desc'
)}}

What do y'all think?

chalk up another win for the humble null

😂
Reflections on the following paths forward:

Move deduplicate to dbt-core
New macro

Move deduplicate to dbt-core
dbt-utils is caught in this funny place where it only supports a small handful of (popular) databases.
For anything inherently part of SQL, we don't need cross-database implementations, 'cause it should "just work".
dbt-utils really works best when it has only a single implementation of a macro and no adapter-specific overrides. It can get caught in difficult places (like this one), when nearly every SQL dialect needs its own implementation. That's where dbt-core + adapters can shine.
Whenever adapter-specific variants are added to a dbt package, it's the perfect moment to stop and consider: there's probably some abstraction that should be in Core!
Adding deduplicate to dbt-core is attractive because many databases can implement the desired behavior. Snowflake, Databricks, and DuckDB all support qualify, and both Postgres and BigQuery have dialect-specific ways to accomplish it.
Since there's not a ubiquitous solution that would work across all dialects, the default implementation in dbt-core would probably be to raise a NotImplementedException.
New macro
To support arbitrary CTEs, there are two main options for databases that don't support something akin to qualify:

accept that row_number() over ... will add a single extra column to your result relation (rn by default, but configurable)
pass in a set of columns manually

Each have their downsides, but the latter seems especially unpleasant.
Overall, the experience for end users of the long tail of dbt adapters is likely to be much better if we move deduplicate to dbt-core rather than wrestle around within dbt-utils.

Some more info...
Support for qualify
Today I Learned (TIL) that BigQuery supports qualify!
So here's a running list (bound to be non-exhaustive):

Snowflake qualify
Databricks qualify
BigQuery qualify
DuckDB qualify
H2 qualify

Included the last one just 'cause I really appreciate @fatalmind database writings like the "Can I use..." series that tipped me off to BigQuery's support on this.
Only in dreams
If all data platforms supported both the row_number() window function and select * exclude, then I think we could construct a one-size-fits-all deduplicate. But in a world where qualify is ubiquitous, exclude isn't necessary and the solution is more direct.

Qualify is supported in Redshift now.
I made a PR: https://github.com/dbt-labs/dbt-utils/pull/811

@dbeatty10 can you help review the PR? thank you!


Tag and commit still done on error

bump-my-version version: 0.23.0
Python version: 3.9-3.12
Operating System: Windows

Description
I just integrated bump-my-version as a pre-push to main and one very strange behaviour is that when bmv fails it still commit/tag
What I Did
> git push
bump-version.............................................................Failed
- hook id: bump-version
- exit code: 1

Usage: bump-my-version bump [OPTIONS] [ARGS]...

 Try 'bump-my-version bump -h' for help
+- Error ---------------------------------------------------------------------+
| Did not find '__version__ = "0.1.4"' in file: '__init__.py'                 |
+-----------------------------------------------------------------------------+

Error during version bump or push: Command '['bump-my-version', 'bump', 'patch']' returned non-zero exit status 2.

> git log                                                                                (4⇡)
commit fd2d26c8e5555ecc2257f91c4a4bd9e011ce6fd4 (HEAD -> main, tag: v0.1.5)
Author: Britney Spears<EMAIL_ADDRESS>Date:   Fri Jun 21 19:57:36 2024 +0200

    ⬆ Bump version: 0.1.4 → 0.1.5

Is there a flag/option I'm missing to avoid this? If not I'll just chain the commit/tagging myself after bmv so that it's not executed when the later fails.

Ah but the issue with chaining is that I will have to find the value of "new_version"

I'll check into the tagging/committing issue today.

@melMass I need more information since I can't reproduce the error. I created a test at it stops before it gets to the commit and tag part.
Can you include your configuration and add -vv to the bump-my-version call for more verbose information?

Thanks for giving it a look, I'll do a more thoughtful reproduction tomorrow

Sorry for the delay, I'm abandoning the idea of tiding bmv to git hooks as there are multiple layers and it's prone to errors (I had the great idea to experiment with all that on a dirty repo 🤡 , lost a bunch of work).
I will comment here if I ever revisit the idea but after trying I would not recommend it!
Also doing it manually is more logical for me in the end.
Thanks for looking into the issue


flutter doctor immediately closes without any output

Steps to reproduce
I am trying to get Flutter and Dart configured on a Windows 11 development box. I am following the flutter installation documentation and when I run the 'flutter doctor', which is one of the early installation verification steps, the cmd windows simply gets closed without any output. Kind of a lame utility if it does nothing but abort.
Following the documentation, I downloaded the Windows Flutter zip file and placed it into a simple folder, D:\Development\Flutter. I already have git installed (git version 2.46.1.windows.1 which I get from running the git --version command). Git is in my path as is the flutter folders. Either this utility is broken (I can't even get this flutter thing to install!!!)
I am working on a project that it seems as though every avenue has a blocker. dotnet gRPC failed because it could not handle UDP sockets so I had to throw out tons of C#/.NET gRPC code and replace it with go and libp2p. I was trying to use .NET Maui, but the Maui team seems unable to produce anything of quality. I reported bugs over two years ago that are still open. I replaced Maui with Uno, and since it is tied heavily to WinUI 3, it will suffer from the low quality of the WinUI/.NET code. Even Windows is getting dumber by the day where simple things do not work any longer.
Visual Studio has bugs that prevent anything written in xaml with a code-behind file from compiling unless you compile it for the base windows configuration - which is completely unacceptable when writing cross-platform code.
In a nutshell, Microsoft and the rapidly deteriorating quality of everything they touch, has made developing cross-platform apps using .NET tools or Visual Studio 2022 virtually impossible.
I am on a brand-new Windows installation and am simply installing the software I need to do my job.
So I must say that I am SEVERELY disappointed that I cannot even get a base installation of Flutter to work using YOUR 'Get Started' information. What gives?!?! It is as if the software gods are all against me!! Maybe what it is saying is that I need to get my resume updated so I can go apply for that Home Depot or Walmart greeter job because this software gig is going to kill me.
Can someone please give me an explanation as to why the base flutter installation documentation does not provide the correct instructions for getting a Windows dev environment configured?  What gives y'all??? Please help me get flutter installed... per YOUR instructions!!
Expected results
I just want the damned thing to work!!
Actual results
nada...
Code sample
Code sample
[Paste your code here]

Screenshots or Video

Screenshots / Video demonstration
[Upload media here]

Logs
Logs
[Paste your logs here]

Flutter Doctor output

This was from a powershell window. When I run this in a plain cmd window, the Flutter Doctor command simply aborts with no output...

It looks like the logic for this is actually trying to invoke git rev-parse HEAD, rather than checking if git is in your PATH:
https://github.com/flutter/flutter/blob/main/bin/internal/shared.bat#L55-L64
What happens if you run git rev-parse HEAD in your Flutter repo?

@mobynet1
Based on the error you are getting Error: Unable to find git in your PATH, it may be similar to https://github.com/flutter/flutter/issues/123161 that you can check.

Please provide these info as well.

@darshankawar At present, I have no Flutter projects or repos, so I don't think that #123161 applies here. I also own the Flutter installation folder and have full control over it and its subfolders. Since I was only doing a flutter doctor with no flutter repo even created, where is there any reference to a repo that could possibly cause the unsafe error to manifest?
Just to be sure, I tried to run flutter doctor inside the folder that I intend to create a repo and it still aborts.
In short, it could have something to do with what git is trying to do, but what that it is, I don't have a clue. I am suspect of that being the cause. There has to be something else at play here.

It looks like the logic for this is actually trying to invoke git rev-parse HEAD, rather than checking if git is in your PATH:
https://github.com/flutter/flutter/blob/main/bin/internal/shared.bat#L55-L64
What happens if you run git rev-parse HEAD in your Flutter repo?

By "your flutter repo", I mean the Flutter SDK installation (this is the directory the batch script is attempting and failing to run git rev-parse HEAD in)

@christopherfujino I get this: 2663184aa79047d0a33a14a3b607954f8fdd8730

@christopherfujino I get this: 2663184aa79047d0a33a14a3b607954f8fdd8730

Hmm, which is a valid Flutter SDK stable release commit: https://github.com/flutter/flutter/commit/2663184aa79047d0a33a14a3b607954f8fdd8730
this looks like a bug in our windows batch file.
To further help us diagnose the bug, can you share the output of:
Where flutter

And verify this path is the same one you just ran git rev-parse HEAD on?
Oh, and also:
Where git

And verify this is the same terminal environment that you are not able to run flutter doctor from (if for example, you run the terminal in VSCode or Visual Studio, those may add additional paths to the PATH). Thanks!

Hi @mobynet1, have you managed to get this resolved?

Guys, I don't know what you are looking for that I did not provide. The above says everything you need to know.

I was doing a fresh install on a machine that never had flutter or dart installed.
Following your 'Get Started' documentation, at some point after making sure that the prereq's were installed, I simply ran flutter doctor and it just vanished... Look at the screenshot provided!
Git IS installed! That is visible in the screenshot.
flutter doctor says git is NOT in the path...but it plainly is... THAT IS THE ERROR!!

This is nothing that I did or should have to do. If I run your utilities, they should work or at a minimum provide better diagnostic error messages. Your flutter doctor did not see git in the path when it clearly is.

Hi @mobynet1,
This issue was automatically closed after there was no response from you for nearly a month after we requested more information.
We understand that you have git installed and it's on your path, but that's not enough information for us to make progress on debugging your issue. We have hundreds of thousands of other users, many of which are using Windows, who aren't encountering the same issue as you so there's likely some edge case that we're encountering on your particular environment that we can't reproduce or investigate without your help.
As @christopherfujino pointed out, this is likely not necessarily a flutter doctor issue but an issue with flutter.bat, which is a wrapper script that downloads artifacts needed for your particular OS and compiles the actual Flutter CLI tool. This error message is coming from here after we try and run git rev-parse HEAD and fail to get a result. If you could add echo %PATH% to the line before and share the output, we can figure out whether or not the script is executing in the environment that we expect it to be.
Also, as @christopherfujino requested in his last comment, please provide the output of where flutter and where git from the terminal you're encountering this issue in which will hopefully give us some more clues as to what's going on here.
Finally, I'd like to remind you to take a moment and read Flutter's Code of Conduct. We're here to help, but inflammatory or aggressive responses, no matter how frustrating an issue may be, don't help us help you.

I provided enough information for you to test this on your own. The Microsoft MAUI team always wants the end users to do their work debugging their code. This is the same here in this situation. Your product is broken and I opened an issue.

You did not, which is why we asked for more information to help us debug the issue. If this was a widespread issue, we would be flooded with issues reporting this problem. That tells me that there's something specific about your setup that is causing the issue, especially since the script is failing so early in its execution on a command that isn't part of our actual tech stack.

I expect vendors to do their own debugging.  I followed your instructions to the letter and flutter doctor failed. Why can't you simply follow my report to see what is going on? Again, I do not have time to debug your product!

We didn't ask you to fix the issue yourself, but we are not oracles with intimate knowledge of your system's configuration. We regularly ask issue reporters for additional information so we can get enough context to figure out what's going on.

where flutter and where git both found the paths. The flutter doctor could not find git. I closed command windows and reopened them to ensure any and all environment variables were set properly.

It's much more helpful to us if you share the output of these commands.

Here's my PATH variable:
`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/steve/.dotnet/tools:/usr/local/go/bin:/usr/lib/dotnet:/home/steve/dotnet:/home/steve/development/flutter/bin:'

Thank you for sharing your PATH, it provides some more clues as to what's going on. This is a Unix styled PATH, not one I would expect to see on Windows.
Do you have WSL installed or git bash? If so, I'm wondering if your terminal is running flutter (the Bash script) instead of flutter.bat. This could explain why where git gives an actual result when run from your Powershell instance but it isn't found by the flutter script. Can you try running flutter.bat doctor or D:\Development\Flutter\bin\flutter.bat doctor and see if that works?

Trying to get Dart/Flutter installed should be seamless and straight forward, but it wasn't. If I download something from GitHub or a vendor, it should not be a stretch to expect it to work out of the box.

I agree 100%, this should just work out of the box and that's the experience we strive to provide. I understand that this is frustrating but, unfortunately, software engineering is hard and it's impossible to predict and handle every edge case, particularly when you're shipping code to run on hundreds of thousands of slightly different but unique environments.

I reported an issue that I felt you would be interested in, but if not that does not bode well about your attitude towards your product's quality. I will stop reporting issues if you are not invested enough to try to do the research necessary to get an issue resolve.

I don't see where we've indicated in this thread that we're not interested in helping you resolve this issue given we've responded to each of your comments. I'm happy to try and help you out here, but it will require some back and forth discussion to resolve. If not, we can leave this issue closed.

I am doing my development on a Linux (Ubuntu 24.04) VM...
On Wed, Nov 13, 2024 at 12:51 PM Ben Konyi @.***> wrote:

I provided enough information for you to test this on your own. The
Microsoft MAUI team always wants the end users to do their work debugging
their code. This is the same here in this situation. Your product is broken
and I opened an issue.

You did not, which is why we asked for more information to help us debug
the issue. If this was a widespread issue, we would be flooded with issues
reporting this problem. That tells me that there's something specific about
your setup that is causing the issue, especially since the script is
failing so early in its execution on a command that isn't part of our
actual tech stack.

I expect vendors to do their own debugging.  I followed your instructions to the letter
and flutter doctor failed. Why can't you simply follow my report to see
what is going on? Again, I do not have time to debug your product!

We didn't ask you to fix the issue yourself, but we are not oracles with
intimate knowledge of your system's configuration. We regularly ask issue
reporters for additional information so we can get enough context to figure
out what's going on.

where flutter and where git both found the paths. The flutter doctor
could not find git. I closed command windows and reopened them to ensure
any and all environment variables were set properly.

It's much more helpful to us if you share the output of these commands.

Here's my PATH variable:

`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/steve/.dotnet/tools:/usr/local/go/bin:/usr/lib/dotnet:/home/steve/dotnet:/home/steve/development/flutter/bin:'
Thank you for sharing your PATH, it provides some more clues as to what's
going on. This is a Unix styled PATH, not one I would expect to see on
Windows.
Do you have WSL installed or git bash? If so, I'm wondering if your
terminal is running flutter (the Bash script) instead of flutter.bat.
This could explain why where git gives an actual result when run from
your Powershell instance but it isn't found by the flutter script. Can
you try running flutter.bat doctor or D:\Development\Flutter\bin\flutter.bat
doctor and see if that works?
Trying to get Dart/Flutter installed should be seamless and straight
forward, but it wasn't. If I download something from GitHub or a vendor, it
should not be a stretch to expect it to work out of the box.
I agree 100%, this should just work out of the box and that's the
experience we strive to provide. I understand that this is frustrating but,
unfortunately, software engineering is hard and it's impossible to predict
and handle every edge case, particularly when you're shipping code to run
on hundreds of thousands of slightly different but unique environments.
I reported an issue that I felt you would be interested in, but if not
that does not bode well about your attitude towards your product's quality.
I will stop reporting issues if you are not invested enough to try to do
the research necessary to get an issue resolve.
I don't see where we've indicated in this thread that we're not interested
in helping you resolve this issue given we've responded to each of your
comments. I'm happy to try and help you out here, but it will require some
back and forth discussion to resolve. If not, we can leave this issue
closed.
—
Reply to this email directly, view it on GitHub
https://github.com/flutter/flutter/issues/156922#issuecomment-2474758786,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5WEZV72EHLXGLXZS44TENT2AO3W5AVCNFSM6AAAAABQAG2CG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZUG42TQNZYGY
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Hello, I am on the same boat and its been a good 8 hours I spent trying to get this up and running.
Starting to think this could be a bug.  Flutter location extracted to: C:\flutter
I'm using Windows 11 64 bit, and followed the installation steps and still get this message when running flutter doctor.
Error: Unable to find git in your PATH.
Like the OP, when I used the regular command prompt, it just spits out the error and closes the window.   If I use PowerShell 5.1, it just shows the error and a new prompt, window stays open.
I have git installed and running where git shows:
C:\flutter\bin\mingit\mingw64\bin\git.exe
(I tried every git path possible...I tried this one along with:
C:\Program Files\Git\bin
C:\Program Files\Git\cmd
C:\flutter\bin\mingit\cmd

I downloaded the most up to date git windows 64 bit version (2.42) and everything seems to be fine.
When running the git rev-parse HEAD:
PS C:\flutter> git rev-parse HEAD
dec2ee5c1f98f8e84a7d5380c05eb8a3d0a81668

Running where flutter:
[C:\flutter\bin\flutter
C:\flutter\bin\flutter.bat]
For my User's windows path environment variable, it includes:
flutter bin, git paths.
I don't believe this is a Path issue now and something to do with flutter.

Hello, I am on the same boat and its been a good 8 hours I spent trying to get this up and running.
Starting to think this could be a bug. Flutter location extracted to: C:\flutter
I'm using Windows 11 64 bit, and followed the installation steps and still get this message when running flutter doctor.
Error: Unable to find git in your PATH.
Like the OP, when I used the regular command prompt, it just spits out the error and closes the window. If I use PowerShell 5.1, it just shows the error and a new prompt, window stays open.
I have git installed and running where git shows: C:\flutter\bin\mingit\mingw64\bin\git.exe (I tried every git path possible...I tried this one along with:
C:\Program Files\Git\bin
C:\Program Files\Git\cmd
C:\flutter\bin\mingit\cmd

I downloaded the most up to date git windows 64 bit version (2.42) and everything seems to be fine.
When running the git rev-parse HEAD:
PS C:\flutter> git rev-parse HEAD
dec2ee5c1f98f8e84a7d5380c05eb8a3d0a81668

Running where flutter: [C:\flutter\bin\flutter C:\flutter\bin\flutter.bat]
For my User's windows path environment variable, it includes: flutter bin, git paths.
I don't believe this is a Path issue now and something to do with flutter.
I also ran flutter.bat doctor with the same results.

Thanks for the update and details @huycozy! That's very helpful!
Since this issue was closed by the reporter, let's continue this investigation over on #159018.

@bkonyi I think you meant @atang08 😄. I only quoted and linked the new issue to this issue.

@bkonyi I think you meant @atang08 😄. I only quoted and linked the new issue to this issue.

Yes I did 😅 oops!

This should be resolved by https://github.com/flutter/flutter/pull/159424.


crash game, pls help

My friend try to play with us in minecraft 1.19.2 on fabric 14.14, but his game crashed if anybody drob item. Pls help me and my friend with it. <3
crash-2023-03-26_18.17.20-client.txt
crash-2023-03-26_18.12.58-client.txt
crash-2023-03-26_18.11.16-client.txt

What made you conclude that this issue is related to Alloy Forgery?
None of the crash reports you sent mention Alloy Forgery at all, they rather put the blame on DropLight or Porting Lib (used by mods like Create Refabricated)

As this is is both stale and seemingly completely unrelated to Alloy Forgery, I'll be closing it. If you have more to say, feel free to comment again and potentially re-open it
Cheers


Any plans to implement SAML authentication ?

Any road map for supporting SAML authentication like github ?

Hi @maneeshmp. To be honest, we haven't thought about this. As far as I can see, it shouldn't be too hard to implement this thanks to Devise. That being said, we are quite busy planning the features for Portus in the next milestone. Honestly, it looks like there's already a ton of other more urgent stuff to be implemented, so this will have low priority for us.
Thus, I'm going to keep this issue open, as a reminder, but we make no guarantees on when this is going to get fixed.

ok, thanks

I think that SAML authentication would be important for a lot of users because i think that most people will use portus as a part of their Development/Deployment Stack.
SAML auth would make it easy to integrate authentication with a SAML provider like gitlab.
For example to integrate gitlab -> gitlab-CI -> portus -> etc. in a workflow.

I guess, it means SAML authentication on the web frontend
I would support this request as well.
Most of our internal web application use single sign on so portus should integrate as well.