New Relay public thread - Q&A and Issues discussions

Hello folks, this issue is open to any questions or problems regarding the new relay implementation.

Status information to confirm relay usage:
Peers detail:
 relay-test-ip-172-20-1-178-rly.netbird.selfhosted:
  NetBird IP: <IP_ADDRESS>
  Public key: CdRpcUnzq2LM9v97VnU7JiiqE0Y4wXp379mXju0efjk=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://relay-eu1.stage.netbird.io <--------------- indicates the relay used to connect to the remote peer
  Last connection update: 2 seconds ago
  Last WireGuard handshake: 3 seconds ago
  Transfer status (received/sent) 92 B/180 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 relay-test-ip-172-20-14-148.netbird.selfhosted:
  NetBird IP: <IP_ADDRESS>
  Public key: bhSrOMLvN+5cMnjWyL4gB+o9En2a1AvAGWNB5N+gEGw=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): <IP_ADDRESS>:51820/<IP_ADDRESS>:51820
  Relay server address: rels://relay-eu2.stage.netbird.io. <--------------- indicates the relay used to connect to the remote peer ( there is a bug which this needs to be cleaned after P2P connection)
  Last connection update: 2 seconds ago
  Last WireGuard handshake: 3 seconds ago
  Transfer status (received/sent) 92 B/180 B
  Quantum resistance: false
  Routes: <IP_ADDRESS>/32
  Latency: 28.5755ms

OS: darwin/arm64
Daemon version: 0.29.0
CLI version: 0.29.0
Management: Connected to [https://test.stage.netbird.io:443](https://test.stage.netbird.io/)
Signal: Connected to [https://signal.stage.netbird.io:443](https://signal.stage.netbird.io/)
Relays:
  [stun:test.stage.netbird.io:3478] is Available
  [turn:test.stage.netbird.io:3478?transport=udp] is Available
  [rels://relay-eu1.stage.netbird.io] is Available.    <--------------- indicates the relay used by your local client (the home relay)
Nameservers:
  [<IP_ADDRESS>:53, <IP_ADDRESS>:53] for [.] is Available
FQDN: maycons-macbook-pro-2-1.netbird.selfhosted
NetBird IP: <IP_ADDRESS>/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 2/2 Connected

Hi,
I have some questions about the new relay which are not clear to me.

In the release notes you wrote "We are moving away from the TURN relay (coturn) to our own relay implementation based on WebSocket".
If I take that literally this means that "only" the TURN part of coturn gets replaced but not the STUN part. Is this correct and the release only the first step to replace coturn completely or is the STUN part also already replaced with the new relay?
In the example mentioned above which indicates the relay used it is I guess in a secured version active but in the release notes only this part is mentioned: "Addresses": ["rel://:"]
Can you enable TLS in the new relay and if yes how? Or is this something for a future release?
I am using Traefik as a reverse proxy and also have implemented Netbird like described in the documentation which works well. I am missing documentation around the new relay and a reverse proxy.

Thanks in advance and also many thanks for your awesome work in building this great software stack!

@landmass-deftly-reptile-budget:

Stun is still going to be required for the P2P discovery. Also, for retro-compatibility, TURN is still required.
The supported URLs are rel:// and rels://, where rels is used for TLS connections. Like signal and management, the relay have Let's Encrypt support, and you can use the environment variables below to enable it:

NB_EXPOSED_ADDRESS=rels://relay.example.com:443  # update the port configuration to match it
NB_LETSENCRYPT_DOMAINS=relay.example.com # should match the exposed address
NB_LETSENCRYPT_DATA_DIR=/etc/letsencrypt # mount this directory for persistency
<EMAIL_ADDRESS>#NB_LETSENCRYPT_AWS_ROUTE53=true # in case you want to use route 53 for issuing the certificate

It also supports certificate files with:
NB_TLS_CERT_FILE=/etc/certificates/cert.crt
NB_TLS_KEy_FILE=/etc/certificates/cert.key

Once this is done, add the exposed address to the management.json file and restart the file.

Relay should work fine behind traefik. We are missing the configuration, but the traffic to the service can be routed with either a domain or with the /relay path prefix.

Hello, I have 2 questions. I am undecided whether to upgrade or not.

I don't fully understand the new Relay Feature. How will it benefit us?
What was the reason to switch to our own relay application? Was there something that the existing system did not meet?

Is it okay to update to 0.29.0 without actually running the new relay image and changing management.json?

For new relay to work is there any new openwrt package released?

Is it okay to use same domain for management, signal, coturn and relay?
Example: If i use domain netbird.domain.com and i want to use this domain for all services but with different port is that okay?

Also, for retro-compatibility, TURN is still required.

If I don't care about old clients, I can ignore TURN completely, right?
Otherwise, this sounds very promising, especially with Kubernetes, the port ranges of TURN have always made the setup a bit more complex. I will definitely give it a try and report back.
STUN will continue to be used in the future?

Replace PORT and DOMAIN according to your deployment.

I have used the automatic setup script, so I am probably using the default values for ports, so what do I need to specify here for PORT in the compose file?

Replace PORT and DOMAIN according to your deployment.

I have used the automatic setup script, so I am probably using the default values for ports, so what do I need to specify here for PORT in the compose file?

It can be found in the setup.env file.
The default port is 33080

Hello,
Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine.
upstream relay-upstream {
    server <IP_ADDRESS>:33080;
}

[...]

# Proxy Relay http endpoint
    location /relay/ {
        proxy_pass http://relay-upstream/relay;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }

try add these:
      proxy_set_header Host            $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_cache_bypass $http_upgrade;

and delete the directive:
proxy_set_header Host $host;

Hello, I have 2 questions. I am undecided whether to upgrade or not.

I don't fully understand the new Relay Feature. How will it benefit us?
What was the reason to switch to our own relay application? Was there something that the existing system did not meet?

@ismail0234 some of the benefits of the new relay over Coturn:

More efficient relay connection for multiple peers:  The ICE mode with TURN opens a connection with the TURN server for each peer connection. That consumes more resources on the client and on the coturn server.

The NetBird connection with the new relay is up to 15% faster than coturn.

The service is easier to run on self-hosted environments, since need to configure a single port.

Built-in TLS/SSL support

The main idea is to have a more efficient relay system for NetBird. Turn/Coturn is a really good system for short-term connections. As a connection via VPN usually lasts many hours or days, we need a more efficient system that can easily be scaled.

Is it okay to update to 0.29.0 without actually running the new relay image and changing management.json?

Yes it is. You don't need to update or configure anything if you don't want. It should be fully compatible with older versions of the management.json file.

For new relay to work is there any new openwrt package released?

We will look into updating the openwrt version.

Hello,
Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine.
upstream relay-upstream {
    server <IP_ADDRESS>:33080;
}

[...]

# Proxy Relay http endpoint
    location /relay/ {
        proxy_pass http://relay-upstream/relay;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }

try add these:
      proxy_set_header Host            $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_cache_bypass $http_upgrade;

and delete the directive:
proxy_set_header Host $host;

I added them but I am still getting the same error. I don't know if it is of any help but this is what I added to the docker-compose.yml file:
# Relay
  relay:
    image: netbirdio/relay:latest
    restart: unless-stopped
    environment:
    - NB_LOG_LEVEL=info
    - NB_LISTEN_ADDRESS=:33080
    - NB_EXPOSED_ADDRESS=netbird.mydomain.com:443
    - NB_AUTH_SECRET=<MYSECRET>
    ports:
      - <IP_ADDRESS>:33080:33080
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

And this is what I added to management.json:
"Relay": {
        "Addresses": ["rel://netbird.mydomain.com:443/relay"],
        "CredentialsTTL": "24h",
        "Secret": "<MYSECRET>"
},

@MDMeridio001 it seems like you are using nginx for SSL termination too, in that case, try this:
    - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443

and
"Relay": {
        "Addresses": ["rels://netbird.mydomain.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "<MYSECRET>"
},

Assuming a brand new deployment and all clients running 0.29+ where does coturn fit in the picture ? Can we just run coturn with --stun-only if retrocompability is no concern ?

@rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them.

@rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them.

Excuse my confusion, but since you say that you still use STUN for peer discovery, and at the same time Coturn won’t be used when the mobile apps are updated. Does that mean that the STUN service is baked into the new Relay now (or the management service) ?

For new relay to work is there any new openwrt package released?

We will look into updating the openwrt version.

I am updating netbird package against openwrt snapshot for months, and I have no problem so far, in fact I have built the new version 0.29.0 and is working fine, and open a PR https://github.com/openwrt/packages/pull/24950, for now I just see the error 2024-09-10T15:25:09-03:00 INFO [peer: [ REDACTED ]=] client/internal/peer/worker_relay.go:59: Relay is not supported by remote peer, probably because I'm not selfhosting, and from release notes:

Cloud support for the new relay feature is coming soon*.

But I'm not backporting to openwrt 23.05, one of my targets is supported only on openwrt snapshot.

@mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds.

Works like a charm, thanks!

Thanks for improving the relay functionality.
I can't find the relay repo in netbirdio github. Will it be private or closed source?

@marcportabellaclotet-mt
https://github.com/netbirdio/netbird/tree/main/relay

A short example for traefik which is working fine for me:
docker-compose.yml
relay:
    image: "netbirdio/relay:latest"
    container_name: netbird-relay
    restart: unless-stopped
    env_file:
      - relay.env
      - common.env
    labels:
      traefik.enable: 'true'
      traefik.http.routers.netbird-relay.rule: 'Host("netbird.mydomain.com") && PathPrefix("/relay")'
      traefik.http.routers.netbird-relay.entrypoints: websecure
      traefik.http.routers.netbird-relay.service: netbird-relay-service
      traefik.http.services.netbird-relay-service.loadbalancer.server.port: 33080

relay.env
NB_LOG_LEVEL=info
NB_LISTEN_ADDRESS=:33080
NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443/relay
NB_AUTH_SECRET=secret

management.json
"Relay": {
        "Addresses": ["rels://netbird.mydomain.com:443/relay"],
        "CredentialsTTL": "24h",
        "Secret": "secret"
    },

  # Relay   relay:     image: netbirdio/relay:latest     container_name: NetBird-Relay     restart: unless-stopped     environment:     - NB_LOG_LEVEL=info     - NB_LISTEN_ADDRESS=:33080     - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443     - NB_AUTH_SECRET=secret     ports:       - 33080:33080     networks:       - proxynet     logging:       driver: "json-file"       options:         max-size: "500m"         max-file: "2"
    "Relay": {     "Addresses": ["rels://netbird.mydomain.com:443"],     "CredentialsTTL": "24h",     "Secret": "secret"     },
`    # NetBird relay service
location /relay/ {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
    set $upstream_app NetBird-Relay;
    set $upstream_port 33080;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}`

I use SWAG reverse proxy which just bundles nginx and lets encrypt, my config files are above. I'm trying to add the new relay service. When I fire up my docker client/agent I get this error in the logs for it:
2024-09-11T20:43:38Z INFO client/internal/connect.go:251: connecting to the Relay service(s): rels://netbird.mydomain.com:443 2024-09-11T20:43:38Z INFO [client_id: sha-rM7aFoAlqgVFnc6cmZ0EO7c212GA6XlmZ/sUWtZS5c8=] relay/client/client.go:141: connecting to relay server: rels://netbird.mydomain.com:443 2024-09-11T20:43:38Z ERRO relay/client/dialer/ws/ws.go:36: failed to dial to Relay server 'wss://netbird.mydomain.com:443': failed to WebSocket dial: expected handshake response status code 101 but got 404 2024-09-11T20:43:38Z WARN relay/client/manager.go:130: Connection attempt failed: failed to connect to rels://netbird.mydomain.com:443: failed to WebSocket dial: expected handshake response status code 101 but got 404 2024-09-11T20:43:38Z ERRO client/internal/connect.go:253: failed to connect to any relay server: all attempts failed

Good for me using traefik as a proxy (my config is similar to @ptpu's one )
@mlsmaycon according to your sample, I guess I can spawn multiple relay instances for redundancy reasons:
meaning can I have this case:
    "Relay": {
            "Addresses": ["rels://relay-a.mydomain", "rels://relay-b.mydomain"],

and you confirm that I cannot have this case: relay.mydomain resolving to multiple IPs  ?
    "Relay": {
            "Addresses": ["rels://relay.mydomain"],

Good for me using traefik as a proxy (my config is similar to @ptpu's one )
@mlsmaycon according to your sample, I guess I can spawn multiple relay instances for redundancy reasons:
meaning can I have this case:
    "Relay": {
            "Addresses": ["rels://relay-a.mydomain", "rels://relay-b.mydomain"],

and you confirm that I cannot have this case: relay.mydomain resolving to multiple IPs  ?
    "Relay": {
            "Addresses": ["rels://relay.mydomain"],

Hey @heisbrot , you can have both, the first one means that the client will try both endpoints at the same time and use the one that responds first.
The second one implies a single node or a load balancer endpoint, in the first case, that should work fine, but for a LB, the nodes would need to expose their own addresses configured in the expose address configuration and it should either point directly to them or point to the LB with a Host routing rule to ensure that the traffic would go the correct node.

@rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them.

Excuse my confusion, but since you say that you still use STUN for peer discovery, and at the same time Coturn won’t be used when the mobile apps are updated. Does that mean that the STUN service is baked into the new Relay now (or the management service) ? (Would we be ultimately able to remove Coturn from docker compose and the management.json ?) Thank you very much for this new implementation it sounds cool and production friendly

We are studying this option. We have two systems that can hold the stun role, signal and relay, which are both involved in the connection discovery, we will have a decision soon. But the idea of not having coturn in our self-hosted scripts and templates is something that will be applied after 2-3 major (v0.X.0) releases.

For new relay to work is there any new openwrt package released?

We will look into updating the openwrt version.

I am updating netbird package against openwrt snapshot for months, and I have no problem so far, in fact I have built the new version 0.29.0 and is working fine, and open a PR openwrt/packages#24950, for now I just see the error 2024-09-10T15:25:09-03:00 INFO [peer: [ REDACTED ]=] client/internal/peer/worker_relay.go:59: Relay is not supported by remote peer, probably because I'm not selfhosting, and from release notes:

Cloud support for the new relay feature is coming soon*.

But I'm not backporting to openwrt 23.05, one of my targets is supported only on openwrt snapshot.
And to be honest someone open a issue openwrt/packages#24569 (comment) on openwrt repo to backport a new version, I offered my help to the person if he could test it, but I got no response.

Thanks for the contribution @wehagy. I've asked the user again if they have some time to test it out.

@mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds.
In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system.
20 Peers: 200-300 ms 100 Peers 300-600 ms 200 Peers: 500-1000 ms 500 Peers: 1500-3000 ms

@ismail0234 We are working on some optimization around our database access. It would be helpful if you can share the exact setup you have, VM size and database you are using, and the API calls you are using to measure it.
Another thing that might affect a self-hosted installation is if you have users in the system that got removed from your IDP, that would cause lots of IDP requests to fetch new data to be cached.

@mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds.
In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system.
20 Peers: 200-300 ms 100 Peers 300-600 ms 200 Peers: 500-1000 ms 500 Peers: 1500-3000 ms

@ismail0234 We are working on some optimization around our database access. It would be helpful if you can share the exact setup you have, VM size and database you are using, and the API calls you are using to measure it.
Another thing that might affect a self-hosted installation is if you have users in the system that got removed from your IDP, that would cause lots of IDP requests to fetch new data to be cached.

@mlsmaycon
I'm using the standard Netbird installation, I haven't made any extra settings, so I guess sqllite is used by default. Users connect to the network via setup-key. Is there a way to find out the database size?
I also don't know what an IDP is. JWT Group sync and User group propagation feature are also disabled. I do not use these features. Right now, I'm kicking all unconnected peers off the network every hour to keep the api running fast, so my problems are completely fixed. As the number of peers in the network increases, api calls become very slow and some calls return 0 http response.
The api calls I use are as follows;

“List all Peers”
“List all Groups”

Since Netbird does not share any information on the client side to find the id of the user in api calls, I need to match the netbird ip. For this, it is necessary to go through all peers and match the correct ip.

@ismail0234 IDP is the identity provider (authentik, Zitadel, Google and others).
NetBird has an integration with the tool of your choice that mainly gets the user name and email and cache so that you can see them as human readable in the dashboard.
Something is really off about your performance. You can try disabling your IDP integration by updating the management.json and setting the following key to "none."
    "IdpManagerConfig": {
        "ManagerType": "zitadel",

to
    "IdpManagerConfig": {
        "ManagerType": "none",

Then you can restart your management server and test. You will see users as IDs during this test.
Also, can you create a ticket for this problem so we can continue there?

@mlsmaycon Where do I create the ticket? Github or slack?

@ismail0234 Github, but feel free to reach out on Slack for a faster iteration.

I followed the above settings for Caddy (as I've used the Zitadel starter script once upon a time), but when a node with the latest client tries to connect I get the following:
2024-09-12T19:35:10Z DEBG client/internal/connect.go:176: connecting to the Management service netbird.mydomain.se:443
2024-09-12T19:35:10Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443
2024-09-12T19:35:10Z DEBG client/internal/connect.go:184: connected to the Management service netbird.mydomain.se:443
2024-09-12T19:35:11Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443
2024-09-12T19:35:11Z DEBG signal/client/grpc.go:81: connected to Signal Service: netbird.mydomain.se:443
2024-09-12T19:35:11Z INFO client/internal/connect.go:251: connecting to the Relay service(s): rels://netbird.mydomain.se:443/relay
2024-09-12T19:35:11Z DEBG relay/client/manager.go:93: starting relay client manager with [rels://netbird.mydomain.se:443/relay] relay servers
2024-09-12T19:35:11Z INFO [client_id: sha-WnVmoeH7RuspQpTopd/8RnZhD9vXJTf3J9VTglSeyGk=] relay/client/client.go:141: connecting to relay server: rels://netbird.mydomain.se:443/relay
2024-09-12T19:35:11Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443
2024-09-12T19:35:11Z ERRO relay/client/dialer/ws/ws.go:36: failed to dial to Relay server 'wss://netbird.mydomain.se:443/relay': failed to WebSocket dial: expected handshake response status code 101 but got 404
2024-09-12T19:35:11Z WARN relay/client/manager.go:130: Connection attempt failed: failed to connect to rels://netbird.mydomain.se:443/relay: failed to WebSocket dial: expected handshake response status code 101 but got 404
2024-09-12T19:35:11Z ERRO client/internal/connect.go:253: failed to connect to any relay server: all attempts failed
Not really sure what I'm missing

@pellz0r Sorry to bother you, but could I see the configuration for your docker-compose.yml and Caddyfile for your relay server?

To confirm:
With the new relay, I still need to have the Coturn instance for STUN at this time?
But I can deploy out multiple relay instances, update the config file accordingly, and it will use those?

Could there be a guide to deploying them on Kubernetes? Thank you!

Could there be a guide to deploying them on Kubernetes? Thank you!

To my understanding, it looks like it's just it's own container. So you'd just spin them up, map the port, and then update the management.json file and rebuild the containers

Relay on k8s (behind a ingress-nginx reverse since it's websockets) :
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  name: netbird-relay-ingress
  namespace: netbird
spec:
  ingressClassName: nginx
  rules:
  - host: netbird.company.com
    http:
      paths:
      - backend:
          service:
            name: netbird-relay
            port:
              number: 80
        path: /relay
        pathType: Prefix
  tls:
  - hosts:
    - netbird.company.com
    secretName: netbird-tls

Service
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: netbird-relay
  name: netbird-relay
  namespace: netbird
spec:
  ports:
  - name: relay
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app.kubernetes.io/name: netbird-relay

Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: netbird-relay
  name: netbird-relay
  namespace: netbird
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: netbird-relay
  template:
    metadata:
      annotations:
        linkerd.io/inject: disabled
      labels:
        app.kubernetes.io/component: relay
        app.kubernetes.io/instance: netbird-relay
        app.kubernetes.io/name: netbird-relay
        app.kubernetes.io/part-of: netbird
    spec:
      containers:
      - env:
        - name: NB_LOG_LEVEL
          value: info
        - name: NB_LISTEN_ADDRESS
          value: :80
        - name: NB_AUTH_SECRET
          valueFrom:
            secretKeyRef:
              key: auth_secret
              name: netbird-relay-authkey
        - name: NB_EXPOSED_ADDRESS
          value: rels://netbird.company.com:443
        image: netbirdio/relay:0.29.2
        imagePullPolicy: IfNotPresent
        name: netbird-relay
        ports:
        - containerPort: 80
          name: relay
          protocol: TCP

The deployment references a netbird-relay-authkey Secret you need to provide it with a key of your choice.

Relay performance question...
I was testing netbird speed using direct connection (opening wg ports) and using relay, and it seems that there is a big performance penalty. Anyone have similars results?
Direct connection : 150Mbit speed
Using Relay: 30 Mbit speed.
I haven't a turn setup, so I can not compare.

Hey @marcportabellaclotet-mt can you check with different MTU configurations for the NetBird interface on both ends of the connection?
Also, can you share which tool you used for the test?

I am using iperf and speedtest.
MTU is 1500 in both sides.
Relay app is deployed as a lxc container.

@marcportabellaclotet-mt
Does the relay have adequate upload/download bandwidth? Since all traffic on a relayed connection flows 'through' it, you're limited by the download/upload of the relay. It may also impact the latency between clients, depending how far the relay is from their point of presence.

I am testing the relay service in the same network where netbird client is hostes, so there is no BW restriction.
My test setup is:

netbird client deployed at main office network.
relay app deployed at main office network
netbird client installed in my laptop at home.

@marcportabellaclotet-mt, the wireguard interface created by NetBird has an MTU of 1280, which can influence the performance and concurrent transfer of other peers' connections in the relay.
A good starting test could be to update your peer's MTU size to 1420 for the wt0 or utun100(macOS) interfaces and test again.

Thanks for answering @mlsmaycon . I will try some debug during the weekend.

Relay compose file
  relay:
    image: netbirdio/relay:latest
    container_name: netbird_relay
    restart: unless-stopped
    environment:
    - NB_LOG_LEVEL=info
    - NB_LISTEN_ADDRESS=:33080
    - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443
    - NB_AUTH_SECRET=secret
    ports:
      - 33080:33080
    networks:
      - proxynet
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

management.json
   "Relay": {
    "Addresses": ["rels://netbird.mydomain.com:443"],
    "CredentialsTTL": "24h",
    "Secret": "secret"
    },

netbird.subdomain.conf
server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name netbird.mydomain.com;

    include /config/nginx/ssl3.conf;

    client_max_body_size 128M;
    client_header_timeout 1d;
    client_body_timeout 1d;

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app netbird_dashboard;
        set $upstream_port 80;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /api {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app netbird_management;
        set $upstream_port 443;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /signalexchange.SignalExchange/ {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;

        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;

        set $upstream_app netbird_signal;
        set $upstream_port 80;
        set $upstream_proto grpc;
        grpc_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /management.ManagementService/ {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;

        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;

        set $upstream_app netbird_management;
        set $upstream_port 443;
        set $upstream_proto grpc;
        grpc_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /relay/ {
        proxy_pass http://netbird_relay:33080/relay;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";

        # Forward headers
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout settings
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
        proxy_connect_timeout 60s;

        # Handle upstream errors
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    }

}

I use SWAG reverse proxy which just bundles nginx and lets encrypt, my config files are above. I'm trying to add the new relay service. When I fire up my docker client/agent I get this error in the logs for it:
UPDATE: the current relay location I have now works.

Thanks @pugnobellum adding the /relay onto the proxy pass location fixed my issue.

@mlsmaycon
It possible to use cloud hosted Netbird (free tier), but self hosted relay?

@mlsmaycon
hi maycom, i want to confirm if running a relay service alongside Coturn will cause connection issues on the user client or maybe connection between service? Currently, many of my users are still using versions below 29

Just if someone uses Caddy as reverse proxy, here is my config, that works fine with the relay container.
# Reverse proxy for Netbird UI, Relay, Signal and API
netbird.example.com: {
    # World accessible GRPC endpoint for all access nodes and clients
    handle /management.ManagementService/* {
        reverse_proxy h2c://<IP_ADDRESS>:33073
    }
    # World accessible relay endpoint
    handle /relay* {
        reverse_proxy http://<IP_ADDRESS>:33080 # Relay
    }

    # Protected management routes
    handle {
        @denied not client_ip <YOUR_ALLOWED_IP_FOR_MGMT>
        abort @denied

        reverse_proxy /api/* http://<IP_ADDRESS>:33073 # Management
        reverse_proxy /* http://<IP_ADDRESS>:8080 # Dashboard Singepage App
        header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        header Referrer-Policy no-referrer
        header Permissions-Policy "geolocation=(), microphone=()"
    }
    log {
	    output file /var/log/caddy/door1.az.dimedis.net.log
    }
}

In the mangement.json I appended:
"Relay": {
        "Addresses": ["rels://netbird.example.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "xyz"
    }

The part
@denied not client_ip <YOUR_ALLOWED_IP_FOR_MGMT>
abort @denied

is, of course, optional. I wanted to make the Management API and UI accessible for a known IP address only.

thankyou @thorstenkramm

Just if someone uses Caddy as reverse proxy, here is my config, that works fine with the relay container.
# Reverse proxy for Netbird UI, Relay, Signal and API

i can confirm this config is working if you using caddy, for me i prefer using relay instead of http://<IP_ADDRESS> and change rel:// to rels:// in management.json
Caddyfile
:80, netbird.domain.com:443 {
    import security_headers
    # Relay
    handle /relay* {
        reverse_proxy relay:33080 # Relay
    }

    # Signal
    reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000
    # Management
    reverse_proxy /api/* management:80
    reverse_proxy /management.ManagementService/* h2c://management:80
    # Zitadel
    reverse_proxy /zitadel.admin.v1.AdminService/* h2c://zitadel:8080
    reverse_proxy /admin/v1/* h2c://zitadel:8080

docker-compose.yaml
 # Relay
  relay:
    image: netbirdio/relay:latest
    restart: unless-stopped
    networks: [netbird]
    env_file:
      - ./relay.env
    ports:
      - 33080:33080
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

management.json
 "Relay": {
        "Addresses": ["rels://netbird.domain.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "secret"
    }

relay.env
NB_LOG_LEVEL=info
NB_LISTEN_ADDRESS=:33080
NB_EXPOSED_ADDRESS=rels://netbird.domain.com:443
NB_AUTH_SECRET=secret

I recommend continuing to use COTURN as the relay. COTURN can be further developed as needed to meet NETBIRD's requirements.
The reasons are as follows:
QoS Control Issues:
Merging all requests into a single TCP connection can make Quality of Service (QoS) difficult to manage effectively.
Complexity of Relay Networks:
The complexity of relay networks is influenced by factors such as firewalls and QoS. Proprietary protocols may encounter unusual issues, such as being blocked, in complex network environments.
Mature Telecom-Grade Solutions:
While many relay solutions exist, currently only TURN can be effectively deployed in telecom-grade solutions. Similar to VXLAN and EVPN, which require collaboration among various vendors to implement based on a standard, TURN stands out as the viable option for reliable relay networks.
Development of Low-Level Communication Protocols:
Developing low-level communication protocols requires many years of accumulation. I believe that adding features at the application layer is much more cost-effective than investing in technologies that may fail during the R&D phase.

What is the benefit of enabling TLS compared to leaving it off?
If I understand this correctly the purpose of the relay is to relay wireguard traffic, which is encrypted already and the relay endpoints are statically configured via management.json
So other than somebody hijacking your DNS to point to a malicious relay while at the same time having stolen the relay secret, why would I enable TLS on the relay?

self host

Same question here. A self-host-able DERP like relay server is very critical for low latency access.

What is the benefit of enabling TLS compared to leaving it off? If I understand this correctly the purpose of the relay is to relay wireguard traffic, which is encrypted already and the relay server endpoints are statically configured via management.json
So other than somebody hijacking your DNS to point to a malicious relay while at the same time having stolen the relay secret, why should I enable TLS?

I wonder essentially the same, what a the recommendation to best secure the Relay service. Normally we put the management cluster behind a Reverse Proxy/API gateway + waf & API protection (with support for grpc). but this configuration will increase latency and create problems for Relay trafic.
so what needs to be protected in the relay service, and what is the official recommendation for security layers to add in production ?
@mlsmaycon
any input guys is welcome. thank you very much in advance

@wehagy can you please let me know where can i find Latest openwrt packages

@wehagy can you please let me know where can i find Latest openwrt packages

@rudradevpal you can find the most up-to-date netbird packages for openwrt using the snapshot version of openwrt. Probably, but I haven't tested it and I might be wrong, you can download the latest .ipk package of netbird from the openwrt snapshot repository and install it on the stable version of openwrt.

I'm experiencing issues with proxying the relay service with caddy, but I am able to do so with nginx, and I was wondering if anyone had configuration advice.
The relay service is using self signed certificates, so the Caddyfile looks like this:
...
        handle /relay* {
                reverse_proxy {RELAY_ADDRESS}:{RELAY_PORT} {
                        transport http {
                                tls
                                tls_insecure_skip_verify
                                read_timeout 86400s
                                write_timeout 86400s
                                keepalive_interval 75s
                        }
                }
        }

And I'm getting this consistently in my logs -- both caddy and the relay service
{"level":"error","ts":1731175600.2423942,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"stream error: stream ID 5; CANCEL"}

{"level":"error","ts":1731175600.2424295,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:10000","duration":0.011144689,"request":{"remote_ip":"<IP_ADDRESS>","remote_port":"26566","client_ip":"<IP_ADDRESS>","proto":"HTTP/2.0","method":"POST","host":"{DOMAIN}:443","uri":"/signalexchange.SignalExchange/ConnectStream","headers":{"User-Agent":["grpc-go/1.64.1"],"Te":["trailers"],"X-Forwarded-For":["<IP_ADDRESS>"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["{DOMAIN}:443"],"X-Wiretrustee-Peer-Id":["{ID}"],"Accept-Encoding":["gzip"],"Content-Type":["application/grpc"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":""}},"error":"reading: stream error: stream ID 5; CANCEL"}

2024-11-09T13:06:53-05:00 ERRO [peer_id: ] relay/server/peer.go:61: failed to read message: failed to get reader: failed to read frame header: EOF

2024-11-09T13:06:53-05:00 DEBG [peer_id: ] relay/server/relay.go:137: relay connection closed


Lines get connected when lineStyle is changed

Hello,
when I draw some lines and afterwards change the lineStyle. The previously draw lines get connected.
This happens in Neko, Windows and Html5 Target. In Flash it works correctly.
Expected result:

Result in Neko, Windows and HTML5

package;

import openfl.Lib;
import openfl.display.Shape;
import openfl.display.Sprite;

class Main extends Sprite {
	public function new() {

		super();

		var shape = new Shape();
		Lib.current.addChild( shape );
		var g = shape.graphics;

		g.lineStyle( 1, 0xFF0000 );
		g.moveTo( 100, 100 );
		g.lineTo( 220, 220 );
		g.lineTo( 10, 240 );
		g.lineTo( 80, 200 );

		g.lineStyle( 1, 0x00FFFF );
	}
}

I'm using
Haxe 3.2.1 or Haxe 3.4
Lime 3.7.2
Openfl 4.7.3
Thanks

Works in Lime 3.7.3 and Openfl 4.8.0
Thanks


Support logging of output operations in job database

Not only full jobs should be logged, but also individual targets

So we eventually need to have "clean" as an additional state.
And a back link to the original job would also be very nice


./retina debug

Retina failures can be obscure. It would be useful to have a debug subcommand which performs sanity checks and validitions on things like kernel eBPF support, and can output the eBPF maps etc for debugging purposes.

will pivot from this https://github.com/microsoft/retina/pull/610#discussion_r1735400479


[otbn] Implement error prioritization

See https://github.com/lowRISC/opentitan/issues/5445 for a bit more background
Where there are multiple errors, some are suppressed as their presence
may depend upon undefined data or overly complex conditions.
In addition a new fatal_bad_err is introduced. This is signalled where
the raw error bits present a combination that cannot be seen in a
correctly functioning OTBN implementation.
Fixes #4996
Signed-off-by: Greg Chadwick<EMAIL_ADDRESS>

I've updated the RTL to address the comments from @imphil. I'll await the conclusion of the discussion @rswarbrick and I are having before addressing the documentation comments as there may be substantial rewrite of the documentation portion of this PR.

We've done a fair amount of work on errors since this PR. And while not all issues are resolved, it feels unlikely we'll merge this PR (maybe take parts from it in a new PR as necessary). Let's close it.


Twitter configuration is incorrect

The Twitter Configuration looks incorrect and incomplete.

fb looks like a copy/paste error.
it says that you really don't need to anything for iOS, but I had to do a few things, like add twitter and twitterauth to LSApplicationQueriesSchemes, and configure the callback handler to be twitterkit-<consumer-key>://

Thanks for pointing it out. Will take a look and fix hopefully this weekend :)


vision begin

Hi Ken,
How are you?  Hope everything's ok on your side!
Have 3 questions here.
Question 1
Have a use case in which I would like to use multiple sikuli statements using vision().
As highlighted in your post: https://github.com/kelaberetiv/TagUI/issues/96
In TagUI, you have
vision begin
vision statement 1
vision statement 2
vision statement 3
vision finish

May I know what's the corresponding syntax in RPA-Python?
Question 2
You commented in the same post that:

For TagUI, Sikuli's integration is done through Python language.

Is it possible that to run sikuli commands directly from within Python?
Question 3
In the same post, you have this sample code:
py begin
execfile('/Users/kensoh/Desktop/v2b')
py finish

What would be the corresponding code using RPA-Python?
If I put the multiple sikuli commands in the file /Users/kensoh/Desktop/v2b, is it possible to run this file directly from RPA-Python?
Thanks in advance for your help!

Hi CK! How are you, how have you been?
Now still bad, but have gotten into the grove (ie work in 2 full-time jobs, 1 is AISG job, 1 is taking care of 2YO baby)
Question 1
In Python there is no flexibility to have such vision begin and finish structure, so it'll either be combining all lines into 1 with ; or do it line by line by breaking up into multiple r.vision() steps. However, you can't do if and for statements this way. Do the if and for loops instead within Python, and call vision step accordingly (see if this works for your scenarios).
r.vision('statement 1; statement 2; statement 3;')

r.vision('statement 1'); r.vision('statement 2'); r.vision('statement 3');

Question 2
SikuliX native scripting language is Jython, that's why you see above can use ; as separator btw lines. However Jython isn't Python, so you can't actually import some sikuli library and use sikuli commands directly. There is a WIP Python package by SikuliX creator/maintainer but it is not yet released - https://github.com/RaiMan/sikulix4python
Question 3
The execfile('filename.py') is a Python 2 function to run a Python script, for Python 3 it is exec(open("filename.py").read()). There is also a 2nd parameter to specify whether variable scope is global or local. However both won't work in your usage scenario of running SikuliX commands, because SikuliX while is using Jython language, it is not a Python package that you can run within Python. So running a Python script that has SikuliX commands won't work.

Btw CK, I've just found a good solution on the Mac Retina display problem from issue #170 -

For mac computers with Retina display, the images captured are at double resolution. Running below in your Python script will auto-scale your supplied images down to the actual resolution to do visual automation. I don't have Retina display, but have tried and tested successfully with a fake double-sized image.
r.vision('Settings.AlwaysResize = 0.5')

r.vision('statement 1; statement 2; statement 3;')

r.vision('statement 1'); r.vision('statement 2'); r.vision('statement 3');

Hi Ken,
For your info, both worked!
I actually wanted to use multiple vision statements because I want to click images in the second monitor.  I googled.  It seems I can achieve these with multiple vision statements.
But with your solution from #252, you have solved my problem with a one-liner:
r.vision('Screen(1).click("d:/tagui/img1.png")')

So no need multiple vision statements now.  But it's good to know this in case I need to use this later.
Thanks a lot, Ken!


knife bootstrap on Windows always prepends drive letters to file_cache_path

Description
When running knife bootstrap on Windows, knife seems to always prepend a drive letter to the file_cache_path configuration option when it generates the client.rb for the remote node. *nix nodes do not like this.
Chef Version
knife 16.9.20
chef-client 14.12.9
Platform Version
knife: MS Windows 10 Professional
chef-client: CentOS 7
Replication Case
Pretty straightforward - attempt to run knife bootstrap from a Windows machine against a *nix node, using any custom value for file_cache_path, e.g.:
knife bootstrap XXX.XXX.XXX.XXX -N NODE.NAME.FQDN -U root -r "recipe[basic_linux_cookbook]" -E MY_ENVIRONMENT --config-option file_cache_path="/tmp/chef/cache"
Client Output (truncated)
The bootstrap process fails with the following error:
[2021-02-02T16:40:26+00:00] FATAL: Stacktrace dumped to /C:/tmp/chef/cache/chef-stacktrace.out
[2021-02-02T16:40:26+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-02-02T16:40:26+00:00] FATAL: ArgumentError: different prefix: "/" and "C:/tmp/chef/cache/cookbooks/basic_linux_cookbook"

Additionally, /etc/chef/client.rb shows:
file_cache_path "C:/tmp/chef/cache"

(Bootstrapping can be completed by simply removing the invalid line from the client.rb on the node and re-running chef-client, but... that adds some extra steps to the provisioning process.)

This issue will be resolved once fix for issue  https://github.com/chef/chef/issues/10367 is at place. The fix is pending on some response on the discussion - https://github.com/chef/chef/pull/11189#discussion_r600197351

Added another work around to the similar issue: https://github.com/chef/chef/issues/10367#issuecomment-840774181

@skylerto  The PR resolves the issue of prep-ending drive letter to file-cache-path but right now file_cache_path is set to file_cache_path "/var/chef/cache"  irrespective of the value given in flag config-option file_cache_path. So this issue is partly solved as of now

Commit by @skylerto  https://github.com/chef/chef/pull/11592 seems to be resolving this issue as well. Once merged both can be closed.

@jfcantu  I was also getting same errors. Updated the knife.rb file with below mentioned variables and it worked:
file_cache_path          nil
file_backup_path         nil

This is a version specific issue. On another workstation having different chefdk version(older version), same cookbook is working fine.
My workstation version details :
C:\chef-starter\chef-repo> chef -v
ChefDK version: 4.9.7
Chef Infra Client version: 15.12.22
Chef InSpec version: 4.21.3
Test Kitchen version: 2.5.2
Foodcritic version: 16.3.0
Cookstyle version: 5.23.0

Fixed by https://github.com/chef/chef/pull/11592


Dark to Light Mode transition renders slider image invisible (white)

There is a specific issue that presents itself when the system in is Dark Mode when an application that uses MacControlCenterUI is run.
When the app is run, the controls initially look correct regardless of Light/Dark Mode.

Set system to Dark Mode
Launch the demo app in this package
Set the system to Light Mode
The slider becomes white which renders it invisible against the white background of the slider itself
Set system back to Dark Mode
The image reverts back to the correct dark-gray color it initially started with when the app launched in Dark Mode

All other controls are unaffected.

Also the slider background does not update when changing between Dark and Light mode.


Authentication failure in RESP3

When trying to connect to a password-protected Redis instance using RESP3, I get the following error:
NOAUTH HELLO must be called with the client already authenticated, otherwise the HELLO AUTH <user> <pass> option can be used to authenticate the client and select the RESP protocol version at the same time

It works just fine if I turn off RESP3 and go back to RESP2.
I've devised a quick patch that seems to fix this problem by ensuring the client is authenticated before attempting to switch protocols:
diff --git a/src/protocol/connection.rs b/src/protocol/connection.rs
index f936197..5565039 100644
--- a/src/protocol/connection.rs
+++ b/src/protocol/connection.rs
@@ -332,8 +332,9 @@ pub async fn create_authenticated_connection_tls(
   let socket = TcpStream::connect(addr).await?;
   let tls_stream = tls::create_tls_connector(&inner.config)?;
   let socket = tls_stream.connect(domain, socket).await?;
-  let framed = switch_protocols(inner, Framed::new(socket, codec)).await?;
+  let framed = Framed::new(socket, codec);
   let framed = authenticate(framed, &client_name, username, password, inner.is_resp3()).await?;
+  let framed = switch_protocols(inner, framed).await?;
   let framed = select_database(inner, framed).await?;

   client_utils::set_client_state(&inner.state, ClientState::Connected);
@@ -360,8 +361,9 @@ pub async fn create_authenticated_connection(
   let username = inner.config.read().username.clone();

   let socket = TcpStream::connect(addr).await?;
-  let framed = switch_protocols(inner, Framed::new(socket, codec)).await?;
+  let framed = Framed::new(socket, codec);
   let framed = authenticate(framed, &client_name, username, password, inner.is_resp3()).await?;
+  let framed = switch_protocols(inner, framed).await?;
   let framed = select_database(inner, framed).await?;

   client_utils::set_client_state(&inner.state, ClientState::Connected);

After applying this patch, it works as intended. If this is deemed to be an appropriate solution, I am happy to send a PR, or you can apply it yourself if it is more convenient.

Thanks for finding this @yrashk. I have a similar change on one of my working branches for the next major version, but it's diverged too much for me to just merge that. Feel free to make a PR, otherwise I can make one this weekend.
I'll most likely take this change, https://github.com/aembke/fred.rs/pull/53, and https://github.com/aembke/fred.rs/pull/51 and release them as 5.1.1 this weekend.

@aembke I sent PR #55 with my patch. Hoping to see 5.1.1 soon!

This is merged and released in 5.2.0. Thanks @yrashk.


[BEAM-14143] Simplifies the ExternalPythonTransform API

Updates the API so that users can just specify arguments and/or keyword arguments instead of Row objects.
Doc: https://docs.google.com/document/d/164rn4otqtQA-8QoLlaoGunUDjLSBBvS2DuR0bn344MQ/edit

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Thanks. PTAL.

cc: @ihji

Thanks. PTAL.

(blocked by https://issues.apache.org/jira/browse/BEAM-14148)


JSON MULTILINE seems not an option in Format now

[I am not able to see the JSON MULTILINE option, how to select it]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 448ddad1-243c-af54-3395-7d022f83dff7
Version Independent ID: 09e32c75-831b-876e-c33a-9cd5a83500b3
Content: Ingest data from IoT Hub into Azure Data Explorer
Content Source: data-explorer/ingest-data-iot-hub.md
Service: data-explorer
GitHub Login: @shsagir
Microsoft Alias: shsagir

@javierdve
Thanks for your feedback! I've assigned this issue to the author who will investigate and update as appropriate.
@shsagir
Could you Please check and add your comments on this and update as appropriate.

Hi @javierdve
Please see this document https://learn.microsoft.com/en-us/azure/data-explorer/ingest-json-formats?tabs=kusto-query-language#the-json-format
In particular this note:
When ingesting using the ingestion wizard, select Ignore data format errors, to ingest data in JSON format. If you leave this check box unselected, the data will be ingested in multijson format. For more information, see edit the schema in the ingestion wizard.

#please-close


Wrong yggtorrrent URL

Yggtorrent has recently updated their URL to yggtorrent.pe, it should be updated.

The change is available on the last version (2.0.10).
Thanks @Vardiak.


[6.x] Adds the ability to validate artisan command arguments/options

This is a second attempt at introducing the ability to define validation rules, attributes, and messages on artisan command classes to allow artisan command argument/option validation as I suggested in the ideas repo:
https://github.com/laravel/ideas/issues/1773
There was a first attempt put up about 2 weeks ago but it doesn't seem like the author is interested in following up in the changes required or must be busy:
https://github.com/laravel/framework/pull/29798
I've addressed the changes suggested there.

@surgiie , Thanks for taking another look at this. I had everything ready to make another pull request just never got round to it.

Just a note, I plan to explore this but I want to tweak a few things on it. I will get back to it before the next major release.

There are a few things that seem a bit off with this implementation:
The hasValidInput method is called quite late.
The execute method is called after $input->validate(), which means Symfony will throw an exception for any missing arguments. So we any required rules for arguments are useless.
Symfony already provides an initialize method that is meant to be overridden. This should be the place where hasValidInput would be called.
This way there is no need to wrap $this->laravel->call([$this, 'handle']) inside an if statement.
The hasValidInput method is a bit indirect.
The method name says "input" but there are no input arguments.
In my opinion it should be named validate and it maybe accept the $input as its only argument.
There is no need to return a boolean, since this method should be called in the initialize method, and it would also be responsible throwing the ValidationException.
Moreover, the exception handler should be responsible for rendering the ValidationException within the renderForConsole method. The displayValidationErrors is not enough because it wouldn't let developers interact with exceptions with PHPUnit, e.g. with handleValidationExceptions or withoutExceptionHandling.
Users should be able to access the validator instance.
Just like the FormRequest does, we should offer a validator method for providing the validator instance, otherwise a createDefaultValidator method should be called. Also, there is a withValidator method that allows developers to attach callbacks after the validation is completed. That would be handy too.

I wonder if this would be better as a package, or if people could just implement this in their apps themselves?

@taylorotwell any chance you made headway on getting back to this?

@taylorotwell Im unable to get back to this at the moment, but ive reopened the PR simply to bring it back to your attention, feel free to close it. If someone else can implement this in a PR then that would be great. I personally would love to see this implemented.

I plan to take no action here.

Still think this would be useful as a native feature. 😭 @ryangjchandler


A user can add a task with specified deadline

to keep track of due tasks

Already can lah

ok, can close already.


fix(ChipGroup): Chip.Icon being overwritten

GitHub Issue (If applicable): closes #557
PR Type
What kind of change does this PR introduce?

Bugfix

What is the current behavior?
Chip.Icon will be overwritten by null if used in a ChipGroup
What is the new behavior?
^ no more.
PR Checklist
Please check if your PR fulfills the following requirements:

[x] Tested code with current supported SDKs
[ ] Docs have been added/updated
[ ] Runtime Tests and/or UI Tests for the changes have been added (for bug fixes / features) (if applicable)
[x] Contains NO breaking changes
[x] Associated with an issue (GitHub or internal)
[x] Commits must be following the Conventional Commits specification.

Other information

/azp run

/azp run


Add missing type definitions

@FaridSafi just noticed that some props weren't on types definition file, so I double checked every prop available on GooglePlacesAutocomplete.propTypes and added missing ones.
Hope to see this merged soon.
Let me know if you need any changes.
Also thanks for the amazing work here.

Hey, I really appreciate your contribution!
I just updated this in the v2 branch a couple of days ago. Hoping to release a beta version tonight.

All these definitions are now available in v2.0.0.
Thanks again for the PR!

@bell-steven awesome. May I ask if version 2.0.0 will break something? In case of breaking changes I still think this PR is useful (also a asap new 1.x version)

In theory it should be a clean upgrade, although there are technically breaking changes.
As the sole maintainer, I don't want to start getting busy with maintaining 2 versions simultaneously.


OCPBUGS-41261: update NoNewRequiredFields comparator and bring additional comparators

from crd-schema-checker to fix validation false positives

/jira refresh

/approve
/lgtm
Assuming tests go green, all good


support null computedMatch prop

I am working on a an alternative Switch component which works with routes that render using children function property. For animation purposes but not using react-transition-group.
Normal Switch component renders only first matched component.
My version of Switch would render all routes but only one will get non-null computedMatch property.
This change is needed to not ignore null value produced by my Switch component.
If you are interested, there is a animated Switch version:
class Switch extends React.Component {
    static contextTypes = {
        router: PropTypes.shape({
            route: PropTypes.object.isRequired
        }).isRequired
    }

    static propTypes = {
        children: PropTypes.node,
        location: PropTypes.object
    }

    componentWillMount() {
        invariant(
            this.context.router,
            'You should not use <Switch> outside a <Router>'
        )
    }

    componentWillReceiveProps(nextProps) {
        warning(
            !(nextProps.location && !this.props.location),
            '<Switch> elements should not change from uncontrolled to controlled (or vice versa). You initially used no "location" prop and then provided one on a subsequent render.'
        )

        warning(
            !(!nextProps.location && this.props.location),
            '<Switch> elements should not change from controlled to uncontrolled (or vice versa). You provided a "location" prop initially but omitted it on a subsequent render.'
        )
    }

    render() {
        const {route} = this.context.router
        const {children} = this.props
        const location = this.props.location || route.location

        const routes = children.slice(0, -1);
        const defaultRoute = children[children.length - 1];

        let found = false;
        const newRoutes = React.Children.map(routes, element => {
            if (!React.isValidElement(element)) return;

            const {path: pathProp, exact, strict, from} = element.props;
            const path = pathProp || from;

            if (found) { // we do not need more active routes, just return unmatched route
                return React.cloneElement(element, {location, computedMatch: null});
            }

            const match = path ? matchPath(location.pathname, {path, exact, strict}) : route.match;

            if (match) {
                found = true;
            }
            return React.cloneElement(element, {location, computedMatch: match});
        });

        newRoutes.push(React.cloneElement(defaultRoute, {location, computedMatch: found ? null : route.match}));

        return <div>
            { newRoutes }
        </div>
    }
}

export default Switch

We shouldn't change behaviors for a single exterior use case. You'll have to find another way to do this.


Unify Content's EntitySystem logging

About the PR

Log things via Log with generated sawmill name rather than an explicitly set one for all Content EntitySystems, except the ones with Rules. In all cases the explicit _sawmill was redundant.
Why / Balance

Bringing consistency to logs (all logs originating from EntitySystem should be recognizible in the logs system.something).
Logs' sawmills match system class name to assist with debugging.
Less likelihood of someone building a new EntitySystem to copy-paste the unnecessary _sawmill and instaed use the appropriate inherited Log instead.

Technical details

This addresses just Content's EntitySystems.
GameRuleSystem and all classes inheriting from it are excluded. I want to include both the calling system name and the rule name in the sawmill name - this however requires engine changes. I thus opted to cut out that part and made this a Content-only PR.
Also, even with the rule changes, the GameRules themselves will be excluded pending antag refactor, because currently they are a hot-hot mess and I'm sure it's preferable by everyone for me to wait for it to cool down before introducing merge conflicts unnecessarily.
Log Sawmill changes:
System class | current master sawmill | this PR's sawmill
VaporSystem.cs: vapor -> system.vapor
DeviceListSystem.cs: devicelist -> system.device_list
ForensicScannerSystem.cs: forensic.scanner -> system.forensic_scanner
MappingSystem.cs: autosave -> system.mapping
MechSystem.cs: mech -> system.mech
LagCompensationSystem.cs: lagcomp -> system.lag_compensation
NpcFactionSystem.cs: faction -> system.npc_faction
EmergencyShuttleSystem.cs: shuttle.emergency -> system.emergency_shuttle
EventManagerSystem.cs: events -> system.event_manager
VendingMachineSystem.cs: vending -> system.vending_machine
SharedAnomalySystem.cs: anomaly -> system.anomaly
SharedDeviceLinkSystem.cs: devicelink -> system.device_link
ThirstSystem.cs: thirst -> system.thirst
Manually tested most (all that I had doubts about) and confirmed that LagCompensationSystem's Log.Level is getting correctly set like this.
Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Log sawmill name changes. Any analyzers/other software that processes logs needs to be adjusted to new sawmills. Full list of changes in PR - section Technical details.

Amazing PR. the _sawmill thing was vv confusing


HRRR Data Not Integrating

I noticed this yesterday on the dev API endpoint but it wasn't an issue until now on the production API. It looks like HRRR data isn't being integrated into the API and it's using the global GFS/GEFS API.
It must be an issue on PirateWeather's end as I see the 16z run of the model on AWS so there should be HRRR data integrated from a previous run being used still.

Oh shoot- good catch here. I was trying to fix a different issue with the grid referencing, but didn't realize it broke this! I'll fix it and push out a new version stat

Ok, v1.3.2 is live! I'm updating the change log now, but the deeply uninteresting issue was that I had the x and y variables reversed, so HRRR threw an issue, which excluded it. So silver lining is that the error checking works I suppose.
Thanks again for bringing this to my attention so quickly!

I see the new version and HRRR data is back so I'm going to close this issue. You should probably check the sources section before you push to live to make sure all the sources are there so this isn't repeated.

@alexander0042 With the 1.5.1 update it shows HRRR as a source but it isn't listed in the sourceTimes block.
  "flags": {
    "sources": [
      "ETOPO1",
      "gfs",
      "gefs",
      "hrrr"
    ],
    "sourceTimes": {
      "hrrr_18-48": "2023-05-27 12:00:00",
      "gfs": "2023-05-27 06:00:00",
      "gefs": "2023-05-27 06:00:00"
    },
    "nearest-station": 0,
    "units": "ca",
    "version": "V1.5.1"
  }

I don't think the currently section is using HRRR data as when I compare the current temp with the first hourly temp it is much lower
  "currently": {
    "time":<PHONE_NUMBER>,
    "temperature": 21.53,
  },
    "hourly": {
    "data": [
      {
        "time":<PHONE_NUMBER>,
        "temperature": 23.66,
      },

Good catch here, and unsurprisingly, same underlying issue as the last time. I was working on returning the lat long of the grid point that the model data came from, but ended up breaking it in a weird way that I didn't notice.
I've corrected it in a v1.5.2, so should be good now!

Can confirm that it's fixed. Thanks.

@alexander0042 Don't forget to update the changelog for the newest versions since it's still only showing up to 1.5.


Can't capture video from a webcam

Hi,
I tried to run the "Capturing Video" sample with videoCapture set to
new VideoCapture(-1);
to receive the video stream from the notebook built in camera.
However, the line
capture.Read(image);
throws the following exception:
"Unknown/unsupported array type"
I'm using openCvSharp3, Windows 7, 64 bit.
Does anybody know what is causing this isssue?
Thank you

please see #142 :bow:

Thanks, it works!


Working with multiple Vala files

Maybe this question is a bit silly, but how can I tell Syntastic that the classes and methods I'm using in one file are implemented in another one?
There are no includes in Vala, as far as I know, as all the files are passed to the compiler, which works everything out.

According to :h syntastic-vala-valac, set g:syntastic_vala_modules or add the list of modules to a // modules: comment?

@lcd047 I tried that (by writing the name of the file, with and without the extension) but it does not work...
In fact it breaks syntastic: no errors are reported any more.

In fact it breaks syntastic: no errors are reported any more.

Please open your test file, set g:syntastic_debug to 3, run the checker, run :mes, and post the output.

So, I've added the file in the modules list as syntext.vala in the main.vala file.
As you can see in line 10, it is added to the valac command preceded by a --pkg.
I think there should be another special line at the top named // files: or // sources: containing the files that are passed directly to the Vala compiler. What do you think?
"main.vala" 238L, 7750C written
syntastic: 41.347268: g:syntastic_version = '3.7.0-237 (Vim 800, Linux, GUI)'
syntastic: 41.347640: &shell = 'bash' (!), &shellcmdflag = '-c', &shellpipe = '2>&1| tee', &shellquote = '', &shellred
ir = '>%s 2>&1', &shelltemp = 1, &shellxquote = '', &autochdir = 0, &shellxescape = ''
syntastic: 41.348743: UpdateErrors (auto): default checkers
syntastic: 41.349682: CacheErrors: default checkers
syntastic: 41.351436: g:syntastic_aggregate_errors = 0
syntastic: 41.351924: getcwd() = '/mnt/data/photoocr/texts/Generate_Synth_Texts/vala'
syntastic: 41.353399: CacheErrors: Invoking checker: vala/valac
syntastic: 41.355777: SyntasticMake: called with options: {'errorformat': '%A%f:%l.%c-%\d%\+.%\d%\+: %t%[a-z]%\+: %m,%
C%m,%Z%m', 'makeprg': 'valac -C --pkg syntext.vala  main.vala'}
syntastic: 41.368631: system: command run in 0.012215s
syntastic: 41.368979: checker output: ['error: Package `syntext.vala'' not found in specified Vala API directories or
GObject-Introspection GIR directories', 'Compilation failed: 1 error(s), 0 warning(s)', '']
syntastic: 41.369912: raw loclist: [{'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '', 'pa
ttern': '', 'text': 'error: Package `syntext.vala'' not found in specified Vala API directories or GObject-Introspecti
on GIR directories'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '', 'pattern': '', 't
ext': 'Compilation failed: 1 error(s), 0 warning(s)'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr':
-1, 'type': '', 'pattern': '', 'text': ''}]
syntastic: 41.370547: getLocList: checker vala/valac returned 1
syntastic: 41.370781: vala/valac raw: [{'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '',
'pattern': '', 'text': 'error: Package `syntext.vala'' not found in specified Vala API directories or GObject-Introspe
ction GIR directories'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '', 'pattern': '',
 'text': 'Compilation failed: 1 error(s), 0 warning(s)'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr
': -1, 'type': '', 'pattern': '', 'text': ''}]
syntastic: 41.371141: quiet_messages filter: {}
syntastic: 41.371340: getLocList: checker vala/valac run in 0.017615s
syntastic: 41.371737: aggregated: {'_sorted': 0, '_name': '', '_owner': 1, '_columns': 1, '_rawLoclist': []}

So, I've added the file in the modules list as syntext.vala in the main.vala file.
As you can see in line 10, it is added to the valac command preceded by a --pkg.
I think there should be another special line at the top named // files: or // sources: containing the files that are passed directly to the Vala compiler. What do you think?
"main.vala" 238L, 7750C written
syntastic: 41.347268: g:syntastic_version = '3.7.0-237 (Vim 800, Linux, GUI)'
syntastic: 41.347640: &shell = 'bash' (!), &shellcmdflag = '-c', &shellpipe = '2>&1| tee', &shellquote = '', &shellred
ir = '>%s 2>&1', &shelltemp = 1, &shellxquote = '', &autochdir = 0, &shellxescape = ''
syntastic: 41.348743: UpdateErrors (auto): default checkers
syntastic: 41.349682: CacheErrors: default checkers
syntastic: 41.351436: g:syntastic_aggregate_errors = 0
syntastic: 41.351924: getcwd() = '/mnt/data/photoocr/texts/Generate_Synth_Texts/vala'
syntastic: 41.353399: CacheErrors: Invoking checker: vala/valac
syntastic: 41.355777: SyntasticMake: called with options: {'errorformat': '%A%f:%l.%c-%\d%\+.%\d%\+: %t%[a-z]%\+: %m,%
C%m,%Z%m', 'makeprg': 'valac -C --pkg syntext.vala  main.vala'}
syntastic: 41.368631: system: command run in 0.012215s
syntastic: 41.368979: checker output: ['error: Package `syntext.vala'' not found in specified Vala API directories or
GObject-Introspection GIR directories', 'Compilation failed: 1 error(s), 0 warning(s)', '']
syntastic: 41.369912: raw loclist: [{'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '', 'pa
ttern': '', 'text': 'error: Package `syntext.vala'' not found in specified Vala API directories or GObject-Introspecti
on GIR directories'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '', 'pattern': '', 't
ext': 'Compilation failed: 1 error(s), 0 warning(s)'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr':
-1, 'type': '', 'pattern': '', 'text': ''}]
syntastic: 41.370547: getLocList: checker vala/valac returned 1
syntastic: 41.370781: vala/valac raw: [{'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '',
'pattern': '', 'text': 'error: Package `syntext.vala'' not found in specified Vala API directories or GObject-Introspe
ction GIR directories'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr': -1, 'type': '', 'pattern': '',
 'text': 'Compilation failed: 1 error(s), 0 warning(s)'}, {'lnum': 0, 'bufnr': 0, 'col': 0, 'valid': 0, 'vcol': 0, 'nr
': -1, 'type': '', 'pattern': '', 'text': ''}]
syntastic: 41.371141: quiet_messages filter: {}
syntastic: 41.371340: getLocList: checker vala/valac run in 0.017615s
syntastic: 41.371737: aggregated: {'_sorted': 0, '_name': '', '_owner': 1, '_columns': 1, '_rawLoclist': []}

What do you think?

It's up to you to pass the proper flags to vala so that the command line makes sense for what you want to do.  As of what those flags might be, I personally have no idea. I know nothing about Vala beyond the name. shrug

The thing is that, the --pkg option is used to pass a certain kind of files (vapi or gir), which are like binding files.
What I have is the source code split into several files, and they must be compiled like:
valac --pkg gio-2.0 --pkg gee-0.8 file1.vala file2.vala -o output_name

Where gio and gee are system libraries. So, as far as I know, if I implement some functionality in file1.vala that is needed by file2.vala, there is no way to tell Syntastic to find file1.vala to construct the proper command line...
Maybe, Syntastic could look for other .vala files in the same directory and add them to the path... Or, as I said, add yet another special line to specify which files the current one depends on, which I think is less error-prone...

Like I said: it's up to you to pass the proper arguments to valac.  Nothing stops you from adding filenames along with options to g:syntastic_vala_valac_args.

I will try that, thank you :)


Update export_from_lobe.py

I was running  export_from_lobe.py and was getting an error in windows....
python dataset/export_from_lobe.py Archi-Mobile-APP C:\Users\Pichau\DATASETS\Archi-Mobile
Traceback (most recent call last):
  File "dataset/export_from_lobe.py", line 13, in <module>
    from dataset.utils import _resolve_filename_conflict
ModuleNotFoundError: No module named 'dataset'

and could fix it by putting the _resolve_filename_conflict function inside the export_from_lobe.py

Ah - I recently added a dependency to another file in the package so you will have to run it in module mode like the other scripts:
python -m dataset.export_from_lobe Archi-Mobile-APP C:\Users\Pichau\DATASETS\Archi-Mobile
I'll update documentation now, and also check out this _resolve_filename_conflict to see what it is doing and if we should merge


PvP timer

Add a timer where players are unable to run commands in the chat (for servers  with a /tp plugin etc.)

You can configure it in configuration file. (config.yml) at the blockedCommands section.

If you need additional help, contact on the official discord support: https://discord.gg/v2rkPb4Q2r


Support Serverless Google Cloud Functions

Any plans to support Google Cloud functions similar to the way Lambda is supported?
If you have an idea of what would need to be done to do this I can potentially help with a PR.
Thanks.

Hey @gregorskii,
Definitely, I haven't got much experience with Google Cloud so I was reluctant to take this on myself. The plumbing work should already be there from the lambda implementation, so it should hopefully just be a case of following the same process.
First step would be to create a package similar to https://github.com/Jeffail/benthos/tree/master/lib/serverless/lambda, which does the execution handler wrapping around the serverless library.
Then add a cmd package similar to https://github.com/Jeffail/benthos/tree/master/cmd/serverless/benthos-lambda
Finally, add the build and archive stages for the cmd package to https://github.com/Jeffail/benthos/blob/master/.goreleaser.yml and it should then get packaged in future releases.
I'm happy to help out with an unfinished PR but I'd need you to do the deployment testing.

Thank you I will review these notes and see if I can help.
Appreciate the direction.

An update on this. It runs quite well in Google Cloud Run as a regular docker image. Cloud Run has similar costs/complexity to Cloud Functions.
I believe Cloud Run suits my use case, up to you if it makes sense to leave this open. I do think others would benefit from it.

Thanks for the update @gregorskii, I'll leave this open as it'd be cool to have as an option at some point.


Blog Post:  pharmaversesdtm is now out!

Blog Post
Relationship with admiral.test and new features

Will you be making this post @kaz462?

yes, will do next week :)

Hey all - did you all still want to post on these packages?  I think it would be very handy.  Post can be short!! :)

@kaz462 I leave it to you to decide - we have announced this in various so I am fine either way.

if it be a "no" @kaz462, then please close the issue.
...but I do think it is worthwhile to have a short post on the data packages!!!

@bms63 Apologies for leaving this issue hanging for a while. I will close it as @manciniedoardo mentioned we have announced the release of two data packages, let's do it for the next release when we work more closely with the NEST team


Please remove or make sure it appears in the correct subdirectory of the source path.

Hey! Previously, I had already created a working folder for this project and the files are automatically stored into the util and simulator folders after I compiled them. However, I am not sure why I am getting this error. Has anyone experienced this issue before? Any help will be appreciated, thank you!

have u checked if all the class files are in the right folders, aka cs2030>>util and cs2030>>simulator.
I only have my class files there and all my java files are in the main working project folder

did u face this problem w ur prev levels? if no i guess u can copy paste the packaging over to here!


Can i add Live Timer Using JavaScript

Please assign this issue to me under hactoberfest

yes you can add it.


 I experience the below issue when trying first step, please help. Thanks.
C:\Python\MAE>python mae_pretrain.py
Files already downloaded and verified
Files already downloaded and verified
Adjusting learning rate of group 0 to 1.2000e-05.
0%|                                                                                | 0/98 [00:00<?, ?it/s]
Traceback (most recent call last):
File "C:\Python\MAE\mae_pretrain.py", line 54, in
predicted_img, mask = model(img)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Python\MAE\model.py", line 141, in forward
features, backward_indexes = self.encoder(img)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Python\MAE\model.py", line 70, in forward
patches, forward_indexes, backward_indexes = self.shuffle(patches)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Python\MAE\model.py", line 33, in forward
patches = take_indexes(patches, forward_indexes)
File "C:\Python\MAE\model.py", line 18, in take_indexes
return torch.gather(sequences, 0, repeat(indexes, 't b -> t b c', c=sequences.shape[-1]))
RuntimeError: gather(): Expected dtype int64 for index
Environtment:
Windows 11
C:\Python\MAE>python --version
Python 3.9.9

Hi @meokey. Sorry for your problem. I have only tested the code on Linux. Your problem seems to be caused by the different default data type for np.arange (on Linux is int64 but on Windows is int32).
I have updated the code to convert the data type explicitly. It should be able to resolve your issue.

Hi @meokey. Sorry for your problem. I have only tested the code on Linux. Your problem seems to be caused by the different default data type for np.arange (on Linux is int64 but on Windows is int32).
I have updated the code to convert the data type explicitly. It should be able to resolve your issue.

it works perfectly! thanks a lot.
However, when I run the 2nd step, it throws another memory issue. It seems that the default numbers of workers and batch_size is too large to my available memory size. Is it possible to determine the number of worker and batch_size dynamically based on available memory size? Thanks.
C:\Python\MAE>python train_classifier.py
Files already downloaded and verified
Files already downloaded and verified
C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\utils\data\dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
cpuset_checked))
Adjusting learning rate of group 0 to 1.0000e-04.
0%|                                                                                | 0/391 [00:00<?, ?it/s]
Traceback (most recent call last):
File "train_classifier.py", line 67, in
logits = model(img)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/bill/python/MAE/model.py", line 161, in forward
features = self.layer_norm(self.transformer(patches))
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\container.py", line 141, in forward
input = module(input)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\timm\models\vision_transformer.py", line 213, in forward
x = x + self.drop_path(self.attn(self.norm1(x)))
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\timm/models/vision_transformer.py", line 189, in forward
attn = (q @ k.transpose(-2, -1)) * self.scale
RuntimeError: [enforce fail at CPUAllocator.cpp:68] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 101451264 bytes. Error code 12 (Cannot allocate memory)

You can just decrease the value of max_device_batch_size utils it fits your device.


Added Face Recognition System project

I have added the requirements file.
Create a virtual environment and install the requirements.
DISCLAIMER: Before installing the requirements, install dlib separately using pip install "path_to_dlib_file\dlib-19.24.99-cp312-cp312-win_amd64.whl"

I have also added an attendance feature along with the face recognition system

@anirudh-248 Make sure to add documentation as README file.
Using streamlit, make a simple UI, where user can upload the image then it can detect the known face and mark attendance accordingly. Once this is done, it'll be merged with the main branch

added the requested changes in the face_rec.py file, which contains the streamlit UI and recognizes faces from a provided image. please let me know if anything else is needed.

@SaiNivedh26 Please review this


MuonUnbiased in misID sample

It is recommended to use MuonUnbiased for all non-Mu samples. Which means we should look at MuonUnbiased flag in the misID ntuples.
So far I've found none. Maybe it's a synthetic variable based on trigger decisions. Need to investigate.

It is possible that MuonUnbiased is just mu_L0MuonDecision_TIS.

It is possible that MuonUnbiased is just mu_L0MuonDecision_TIS.

My understanding is wrong. Muon TIS still requires the Muon trigger to be triggered; it only requires that the particle doesn't trigger it.
We should just duplicate our L0 trigger line. The L0 trigger line makes our analysis unbiased by construction.


SAK-29284 Use features in Maven 3.3.1 to reduce setup.

Maven 3.3.1 allows MAVEN_OPTS to effectively come from the project source code. This means that developers have one less thing to configure when setting up their build.
On most developer machine the heap space options shouldn’t be needed as typically the JVM will limit it’s heap to 1/4 of physical memory, but it’s worth setting because some people may be compiling under VMs with much more limited memory constrains.
The permgen options aren’t included here as they are no longer supported under JDK 8 which is now required for master.

Does this still work if building just one module?

Yes. If you cd kernel; mvn install then it still picks up the options
from the top of the repository.
On 21 May 2015 at 14:12, Steve Swinsburg<EMAIL_ADDRESS>wrote:

Does this still work if building just one module?
—
Reply to this email directly or view it on GitHub
https://github.com/sakaiproject/sakai/pull/553#issuecomment-104273200.

--
Matthew Buckett

Nice one.


When enabling "clean" mode with "-m yes", contig names are changed to "contig_#" by mistake?

When enabling "clean" mode with "-m yes", contig names are changed to "contig_#". This causes mismatch between contig name in .bed data and in SALSA's output file.
Line 47 of correct.py should be: contig2newseq[seq] = input_seqs[seq]?
https://github.com/machinegun/SALSA/blob/8b326ccd4456eb8479d75d44fda1f101cb7c2529/correct.py#L47

Thanks a lot. I have fixed this bug.


JTReg Timeout on arm32: java/security/KeyStore/PBETest.java

https://github.com/AdoptOpenJDK/openjdk-tests/issues/1548

test category, sanity.openjdk.jdk_security1_0
OS/architecture, aarch32_linux
public build SHAs (i.e. java -version output), in Grinder provides JDK_VERSION/JDK_IMPL info and helps identify point at which regression introduced:

11:41:14  openjdk version "1.8.0_232"
11:41:14  OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
11:41:14  OpenJDK Zero VM (AdoptOpenJDK)(build 25.232-b09, interpreted mode)

Created via non-JIT VM for the JDK8-hotspot-arm32 GA release (see linked issue)

13:08:21  STDOUT:
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/openjdk/openjdk-jdk/jdk/test/java/security/KeyStore/keystore.jks
13:08:21  Loaded keystore with 2 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21  Encrypted key entry using: PBEWithSHA1AndDESede (with PBE parameters)
13:08:21  Storing keystore to: PBEWithSHA1AndDESede.p12
13:08:21  Loading keystore from: PBEWithSHA1AndDESede.p12
13:08:21  Loaded keystore with 1 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/openjdk/openjdk-jdk/jdk/test/java/security/KeyStore/keystore.jks
13:08:21  Loaded keystore with 2 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21  Encrypted key entry using: PBEWithSHA1AndDESede (without PBE parameters)
13:08:21  Storing keystore to: PBEWithSHA1AndDESede.p12
13:08:21  Loading keystore from: PBEWithSHA1AndDESede.p12
13:08:21  Loaded keystore with 1 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/openjdk/openjdk-jdk/jdk/test/java/security/KeyStore/keystore.jks
13:08:21  Loaded keystore with 2 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21  Encrypted key entry using: keyProtectionAlgorithm=PBEWithSHA1AndDESede
13:08:21  Storing keystore to: PBE.p12
13:08:21  Loading keystore from: PBE.p12
13:08:21  Loaded keystore with 1 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/openjdk/openjdk-jdk/jdk/test/java/security/KeyStore/keystore.jks
13:08:21  Loaded keystore with 2 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21  Encrypted key entry using: PBEWithHmacSHA1AndAES_128 (with PBE parameters)
13:08:21  Storing keystore to: PBEWithHmacSHA1AndAES_128.p12
13:08:21  Loading keystore from: PBEWithHmacSHA1AndAES_128.p12
13:08:21  Loaded keystore with 1 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/openjdk/openjdk-jdk/jdk/test/java/security/KeyStore/keystore.jks
13:08:21  Loaded keystore with 2 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21  Encrypted key entry using: PBEWithHmacSHA1AndAES_128 (without PBE parameters)
13:08:21  Storing keystore to: PBEWithHmacSHA1AndAES_128.p12
13:08:21  Loading keystore from: PBEWithHmacSHA1AndAES_128.p12
13:08:21  Loaded keystore with 1 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/openjdk/openjdk-jdk/jdk/test/java/security/KeyStore/keystore.jks
13:08:21  Loaded keystore with 2 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21  Encrypted key entry using: keyProtectionAlgorithm=PBEWithHmacSHA1AndAES_128
13:08:21  Storing keystore to: PBE.p12
13:08:21  Loading keystore from: PBE.p12
13:08:21  Loaded keystore with 1 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/openjdk/openjdk-jdk/jdk/test/java/security/KeyStore/keystore.jks
13:08:21  Loaded keystore with 2 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21  Encrypted key entry using: PBEWithHmacSHA224AndAES_128 (with PBE parameters)
13:08:21  Storing keystore to: PBEWithHmacSHA224AndAES_128.p12
13:08:21  Loading keystore from: PBEWithHmacSHA224AndAES_128.p12
13:08:21  Loaded keystore with 1 entries
13:08:21  Retrieved key entry named 'vajra'
13:08:21
13:08:21  Loading keystore from: /home/jenkins/workspace/Grinder/openjdk-tests/open
13:08:21  TEST RESULT: Error. Agent error: java.lang.Exception: Agent 2 timed out with a timeout of 960 seconds; check console log for any additional details

Not seeing this failure reproduce in Grinders (example: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4265) so will close. If it does reproduce on a particular machine, it should be raised as an infra issue (and machine should have ci.role.test tag removed until it can be corrected).
These tests were not excluded using the correct OS-ARCH name (used linux-aarch32 instead of linux-arm), entries in problem lists will be removed as part of #2018.


[TASK] Remove library requirement for liblapack

Issue Description
Is your feature request related to a problem? Please describe.
In the past, liblapack 3.9.0 build 21 is broken, so liblapack 3.9.0 has been fixed to 20. New versions of liblapack 3.9.0 is available, such as build 22, 23, and 24. In this case, the requirement can be removed.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

For Change Control Board: Issue Review
This review should occur before any development is performed as a response to this issue.

[x] 1. Is it tagged with a type: defect or task?
[x] 2. Is it tagged with a priority: critical, normal or minor?
[x] 3. If it will impact requirements or requirements tests, is it tagged with requirements?
[x] 4. If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
[x] 5. Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

For Change Control Board: Issue Closure
This review should occur when the issue is imminently going to be closed.

[x] 1. If the issue is a defect, is the defect fixed?
[x] 2. If the issue is a defect, is the defect tested for in the regression test system?  (If not explain why not.)
[x] 3. If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
[x] 4. If the issue is a defect, does it impact the latest release branch? If yes, is there any issue tagged with release (create if needed)?
[x] 5. If the issue is being closed without a pull request, has an explanation of why it is being closed been provided?

Approved to close via #2378.


Improve caching in ElementwiseKernel

Using local and global dictionary for fast lookup.

Jenkins, test this please

Jenkins, test this please

Jenkins CI test (for commit 3ae24605a293aee86d85d3bc7cd2e65a1ee3afea, target branch master) succeeded!

@okuta Please check this PR: #2474

This PR also uses global dictionary to keep #2474.
https://github.com/cupy/cupy/pull/2688/files#diff-e2a721d3ce2742c7fd3e3e565742b7feR401

OK, I didn't notice @util.memoize.
Please post a benchmark.

benchmark code
before
% python adam_elem2.py
adam     ,    1, 0.017272, 0.017270
adam     ,   10, 0.017024, 0.017023

after
% python adam_elem2.py
adam     ,    1, 0.015939, 0.015937
adam     ,   10, 0.015753, 0.015752

What do the numbers mean? I can't see what util does.

This benchmark is for _get_elementwise_kernel. util pass has not passed.

I wanted to ask the units of the numbers.

util pass has not passed.

It looks each line is generated from within util. Am I missing anything?
Btw, I don't think it's that meaningful to measure _get_elementwise_kernel directly unless it's proven a hotspot.

Oh, I misunderstood. I used this.
The unit is millisecond.

So, the difference is just 65 ns?
Honestly I don't think it's worth complicating the code.

In my understanding 1085 ns?

Sorry, I overlooked the numbers were averaged.
But I wonder how this fix could improve that much. It's contradictory to my previous benchmark in #2474.

Thanks. I tried cupy-pref.
before  ( 2e0515457848afb3cbd173b4d4f6b5dab53f24cc )
userkernel          :     8.064 us   +/- 0.245 (min:    7.801) us

after
userkernel          :     7.607 us   +/- 0.222 (min:    7.389) us

kindly ping @niboshi

Sorry for delay.
Jenkins, test this please

Jenkins CI test (for commit 3ae24605a293aee86d85d3bc7cd2e65a1ee3afea, target branch master) succeeded!

LGTM


Context of commit statuses is not really "nice"

Clear and concise description of the problem
That's not a big problem, but currently the context is reported as ci/woodpecker/<event>/<pipeline>. This doesn't look really nice in UIs.
Suggested solution

just use Woodpecker CI as context and put more information into the description
Only report the pipeline name and event as context (this is similar to GitHub Actions) and add more information in the description

Alternative
No response
Additional context
GitHub Actions example:

Woodpecker example:

Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

we  can experiment around a bit - but you have to take in mid, that branch protection roules depend on context be be different
so we realy have to look into each forge if there is a potion to split UI view and actual context

if we have to change the context - it will be breaking - just to mention


LSTM different input and output lengths

I want to create a network with an LSTM layer, where the number of outputs from the LSTM layer is different from the number of its inputs. Is this possible?

OK, so binary classification, for example, on a sequence. Yes, there are two ways to do this. One way, which is not ideal (but works), is just to set Tos = T - 1. In this way, the LSTM outputs the classification on the final time step. Alternatively, you could set Tos=0 and ask the LSTM to predict the same thing for every step of the entire sequence.
The better way, which I highly recommend, is to implement temporal average pooling. I don't have the code for this implemented, but you sum up and average the output of the LSTM for every time step. Then the average is input to a softmax classifier after the final time step. You then would back-propagate by multiplying the error term (d) by the averaging factor, T.

I am very thankful for your reply. I am currently trying the first path and will move to the second way as well.


Rule 'Fields' error

Hi go this error but can't understand it. My variable name is correct, the only one that would need a '_' are the private one but no error/warnings reported on it.
Name should have prefix '_'. (Rule 'Fields').	public AudioClip deathClip;
I think, that you inverse the two conventions.

I have had similar problems with this rule and cannot get OmniSharp to ignore it despite it being in the config.json my server loads.


this.waterline.schema is undefined

I'm trying to use my Sails.js models outside of the sails context. I have one simple model called "Request", and I'm using the sails-disk adapter. Everything works fine inside the Sails app, but when I try and use the model outside the app (I'm running a separate proxy server that needs to save requests passing through it) I get the error below. It seems that this.waterline.schema is undefined for some reason.
Error:
/path-to-app/node_modules/sails/node_modules/waterline/lib/waterline/core/index.js:70
var schemaAttributes = this.waterline.schema[this.identity].attributes;

TypeError: Cannot read property 'request' of undefined

api/models/Request.js
module.exports = {
  identity: 'request',
  schema: true,
  autoCreatedAt: true,
  autoUpdatedAt: true,
  migrate: 'alter',
  attributes: {
    id: {
      type: 'string',
      primaryKey: true,
      required: true
    },

    method: 'string',
    url: 'string'
}

./proxy.js attempt to load sails waterline model in external script (abbreviated)
var uuid = require('node-uuid'),
  Waterline = require('sails/node_modules/waterline'),
  RequestCollection = require('./api/models/Request.js'),
  sailsDisk = require('sails-disk');

RequestCollection.adapter = 'localDiskDb';
var Request = Waterline.Collection.extend(RequestCollection);

new Request({
  adapters: {
    localDiskDb: sailsDisk
  }
}, function(err, collection) {
  if (err) {
    console.error(err);
    return;
  }

  // Save the request
  collection.create({
    id: uuid.v4(),
    method: 'foo method',
    url: 'foo url'
  }).exec();
});

I'm having this issue as well. Using v0.10.18 from NPM.
I poked around in the source code a bit, and I didn't see anything obviously wrong.

Checkout the express example for an example of loading the collections. Also checkout @joshbeitler project Piranha for an easy to use solution.
new Request isn't the correct way to load the collection.

Okay, so both the README.md and waterline-docs (neither of which mentions new Waterline() or initialize) describe an obsolete interface.
What documentation describes basic usage of the current version of Waterline, beyond the examples?