Unable to connect to grps end point using nginx endpoint

Bug description:
I am using the docker-desktop and enabled the kubernets to deploy some of the microservices. as part of my testing, I have deployed a sample grpc service and created the ingress endpoint which is working fine with below configuration.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo-test
labels:
app: demoservice-v10
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-headers: "TenantId,UserId,x-user-agent,authorization,grpc-accept-encoding,grpc-timeout,Access-Control-Allow-Origin,Access-Control-Allow-Credentials,Cache-Control,Access-Control-Allow-Headers,content-type,pragma"
nginx.ingress.kubernetes.io/cors-allow-origin: "https://aigcdnstorageacct.z22.web.core.windows.net,https://hsprdemoapp.azurewebsites.net"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-body-size: 10m
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Frame-Options: DENY";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "Referrer-Policy: strict-origin";
more_set_headers "Permissions-Policy: geolocation=(self)";
more_set_headers "Content-Security-Policy: img-src 'self' data:";
spec:
rules:

host: demtest.<IP_ADDRESS>.nip.io
http:
paths:

path: /
pathType: Prefix
backend:
service:
name: demoservice-v10
port:
name: grpc
tls:

hosts: demtest.<IP_ADDRESS>.nip.io
secretName: tls-secret

Then, i have installed the OSM in local cluster, added the corresponding namespace to the mesh(with proper label and annotation).
Then I have restarted the services, I could see that the envoy proxy is injected for the pod.
I have followed the example given in https://release-v0-11.docs.openservicemesh.io/docs/demos/ingress_k8s_nginx/ to enable the https ingress endpoint with below configuration.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demoosm-test
namespace: demo
labels:
app: demoservice-v10
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_ssl_name "demoservice-v10.uop.cluster.local";
nginx.ingress.kubernetes.io/proxy-ssl-secret: "osm-system/osm-nginx-client-cert"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
spec:
ingressClassName: nginx
rules:

host: demtest-osm.<IP_ADDRESS>.nip.io
http:
paths:

path: /
pathType: Prefix
backend:
service:
name: demoservice-v10
port:
name: grpc

kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
name: demoosm-test
namespace: uop
spec:
backends:

name: demoosm-test
port:
number: 8080
protocol: http
tls:
skipClientCertValidation: true
sources:
kind: Service
namespace: ingress-nginx
name: nginx-ingress-controller
kind: AuthenticatedPrincipal
name: ingress-nginx.ingress-nginx.cluster.local

But, I am not able to get access to the service endpoint using the ingress endpoint.
I am getting below errors from the nginx-controller:
2022/11/09 16:43:28 [error] 2087#2087: *582044 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: <IP_ADDRESS>, server: demotest-osm.<IP_ADDRESS>.nip.io, request: "GET / HTTP/2.0", upstream: "grpc://<IP_ADDRESS>:8080", host: "demotest-osm.<IP_ADDRESS>.nip.io"
Getting below error from the osm-controller:
2022-11-09T22:08:42+05:30 {"level":"error","component":"envoy/lds","time":"2022-11-09T16:38:42Z","file":"inmesh.go:487","message":"Cannot build outbound filter chain, unsupported protocol https for traffic match outbound_ingress-nginx/ingress-nginx-controller_443_https"}
2022-11-09T22:08:42+05:30 {"level":"error","component":"envoy/lds","time":"2022-11-09T16:38:42Z","file":"inmesh.go:487","message":"Cannot build outbound filter chain, unsupported protocol https for traffic match outbound_ingress-nginx/ingress-nginx-controller-admission_443_https"}
2022-11-09T22:08:42+05:30 {"level":"info","component":"envoy/rds","time":"2022-11-09T16:38:42Z","file":"response.go:127","message":"RDS did not fulfill all requested resources (diff: Set{}). Fulfill with empty RouteConfigs."}
And below are trace/debug messages from the envoy proxy running in the pod:
[2022-11-09 16:46:37.881][19][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: new connection accepted
[2022-11-09 16:46:37.881][19][trace][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:66] original_dst: set destination to <IP_ADDRESS>:8080
[2022-11-09 16:46:37.881][19][debug][conn_handler] [source/server/active_stream_listener_base.cc:34] closing connection from <IP_ADDRESS>:60880: no matching filter chain found
I have been trying to chase this problem, but could not succeed, and I can't proceed further.
FYI... I have tried Istio for the same service, it was working without any issues... i guess it might be a problem with osm configuration.
Affected area (please mark with X where applicable):

Install                    [ ]
SMI Traffic Access Policy  [ ]
SMI Traffic Specs Policy   [ ]
SMI Traffic Split Policy   [ ]
Permissive Traffic Policy  [ ]
Ingress                    [ X]
Egress                     [ ]
Envoy Control Plane        [X ]
CLI Tool                   [ ]
Metrics                    [ ]
Certificate Management     [X ]
Sidecar Injection          [ ]
Logging                    [ ]
Debugging                  [ ]
Tests                      [ ]
Demo                       [ ]
CI System                  [ ]

Expected behavior:
The ingress should be able to reach to the service endpoint.
Steps to reproduce the bug (as precisely as possible):
I have explained above.
How was OSM installed?:
OSM was installed using the helm charts with below overrides:
the original file is available https://github.com/openservicemesh/osm/blob/release-v0.11/charts/osm/values.yaml
OpenServiceMesh:
Enable permissive traffic policy mode
enablePermissiveTrafficPolicy: true
Log level for the Envoy proxy sidecar. Non developers should generally never set this value. In production environments the LogLevel should be set to error
envoyLogLevel: trace
Enable the debug HTTP server on OSM controlle
enableDebugServer: true
When enabled, fine grained control over Egress (external) traffic is enforced
Enable egress in the mesh
enableEgress: true
-- Enable mesh-wide HTTPS ingress capability (HTTP ingress is the default)
useHTTPSIngress: true
Anything else we need to know?:
Bug report archive:

Environment:

OSM version (use osm version):
Kubernetes version (use kubectl version):
Size of cluster (number of worker nodes in the cluster):
Others:

Hi,
Any updates on this issue...? anybody is using the nginx with OSM...?
Thanks
Rama.

The following errors are false alarms and can be ignored:
2022-11-09T22:08:42+05:30 {"level":"error","component":"envoy/lds","time":"2022-11-09T16:38:42Z","file":"inmesh.go:487","message":"Cannot build outbound filter chain, unsupported protocol https for traffic match outbound_ingress-nginx/ingress-nginx-controller_443_https"}
2022-11-09T22:08:42+05:30 {"level":"error","component":"envoy/lds","time":"2022-11-09T16:38:42Z","file":"inmesh.go:487","message":"Cannot build outbound filter chain, unsupported protocol https for traffic match outbound_ingress-nginx/ingress-nginx-controller-admission_443_https"}

The below error likely implies a misconfiguration:
[2022-11-09 16:46:37.881][19][debug][conn_handler] [source/server/active_stream_listener_base.cc:34] closing connection from <IP_ADDRESS>:60880: no matching filter chain found

Could you share the bug report using the osm support bug-report command so we can diagnose this further.
Thanks

Hi,
I have tried running the given command below is the output:
PS C:\Users\ramachandraiah.pandu\Downloads\osm-v1.2.2-windows-amd64\windows-amd64> .\osm.exe support bug-report
[+] Created staging dir C:\Users\RAMACH~1.PAN\AppData\Local\Temp\osm_bug_report_1935124464 to generate bug report
[+] Collecting information about meshed namespaces
✓ Finished generating report for meshed namespaces
[+] Collecting information from individual app namespaces
✓ Finished generating report for individual app namespaces
[+] Collecting information from individual app pods
✓ Finished generating report for individual app pods
[+] Collecting information from control plane
✓ Collected report for Pod osm-system/osm-bootstrap-65fdb64f8f-fl4cq
✓ Collected report for Pod osm-system/osm-controller-58cbc6bc89-9qpxx
✓ Collected report for Pod osm-system/osm-injector-86c857c76d-kvk4r
✓ Finished generating report for control plane
[+] Collecting information from individual app namespaces
x Error removing existing bug report file C:\Users\RAMACH~1.PAN\AppData\Local\Temp\3020840240_osm-bug-report.tar.gz
x Error archiving bug report
Error: Error creating bug report: Error generating bug report: remove C:\Users\RAMACH~1.PAN\AppData\Local\Temp\3020840240_osm-bug-report.tar.gz: The process cannot access the file because it is being used by another process.
PS C:\Users\ramachandraiah.pandu\Downloads\osm-v1.2.2-windows-amd64\windows-amd64>
It is failling to generate the .tar.gz file.
But, it created the folder 'osm-bug-report-' folder which I am attaching here...
osm_bug_report.zip

Also, i have attached the envoy proxy log from the target pod (i.e. periodicscript).
envoyproxy_log.zip
In this case, the ingress controller (nginx) pod IP is : <IP_ADDRESS> and periodicscript pod ip: <IP_ADDRESS>. When ingress endpoint is accessed through browser, the request is reaching to envoy proxy in periodicscript, but the envoy proxy is looking for matching filter chain which is not available, hence the connection is closing.
Below is the sample log from envoy proxy.
[2022-11-15 11:34:37.522][26][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:88] tls inspector: new connection accepted
[2022-11-15 11:34:37.522][26][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:175] tls inspector: recv: 746
[2022-11-15 11:34:37.522][26][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: new connection accepted
[2022-11-15 11:34:37.522][26][trace][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:66] original_dst: set destination to <IP_ADDRESS>:8080
[2022-11-15 11:34:37.522][26][debug][conn_handler] [source/server/active_stream_listener_base.cc:34] closing connection from <IP_ADDRESS>:56340: no matching filter chain found

For some reason the bug-report does not contain the dumps from the application namespace and deployments.
Could you try to generate the bug report again by explicitly specifying the application namespace and deployment/pod info as noted below. Also for Windows, you need to specify the zip extension for the output file:
# Generate a bug report for the given namespaces, deployments, and pods
osm support bug-report --app-namespaces bookbuyer,bookstore \
	--app-deployments bookbuyer/bookbuyer,bookstore/bookstore-v1 \
	--app-pods bookthief/bookthief-7bb7f9b98c-qplq4
        --ingress -o bugreport.zip

Hi,  I have ran the below command and generated the bugreport2.zip
.\osm.exe support bug-report --app-namespaces uop --app-deployments uop/periodicscript --app-pods uop/periodicscript-v10-c7c54c8c-2z64v --ingress -o bugreport2.zip
The file is attached here.
bugreport2.zip

One thing to note that the i have tried by modifying the "enableIngressBackendPolicy: true (or) false "  in meshconfig, but nothing is working.

You can ignore below messages from osm-system-controller log.
"Could not list endpoints of the source service osm-system/osm-contour-envoy specified in the IngressBackend uop/periodicscripttest"
I was trying to verify the ingress with contour, but didnt work. ALso, i forgot to remove these backends.

Hi @shashankram, could you please let me know your analysis on this issue...?
Thanks
Rama.

@rcpandula could you clarify the service for which ingress is failing? The bug description indicates it pertains to the  demo/demoservice-v10 service, however the logs are not collected for this service and its deployment. Please collect a bug-report for the corresponding namespace and app deployments as the report provided only has the logs for the app namespace uop.

@rcpandula could you clarify the service for which ingress is failing? The bug description indicates it pertains to the  demo/demoservice-v10 service, however the logs are not collected for this service and its deployment. Please collect a bug-report for the corresponding namespace and app deployments as the report provided only has the logs for the app namespace uop.

Also please collect the logs right after reproducing the issue.

Hi @shashankram, the issue is with the uop/periodicscript-v10 service. THis is the reason, I have collected the bug report for this service.
I have a ingress endpoint periodicscript service and i am trying to access from the browser. The call has reached to envoy side car proxy in periodicscript pod, and the envoy proxy accepted the new connection, but it also looking for the entry in the filter chain which is failing.
I have not created any filter chain for ingress... I am just using the default configuration with "enablePermissiveTrafficPolicy:true" value.
Is this configuration is sufficient to enable inbound traffic to service from external clients ...?
Please let me know what kind of configuration we need to create....
Thanks.

One more thing, i am trying to set '    useHTTPSIngress: true' in osm-mesh-config, but for somereasons it is failling. It is giving 'strict decoding error: unknown field "spec.traffic.useHTTPSIngress"' error.
I am using OSM v1.2.2 release version.

@rcpandula I looked at the periodicscript-v10 service, and unfortunately the logs collected include the invalid Contour configuration you were experimenting with:
apiVersion: policy.openservicemesh.io/v1alpha1
kind: IngressBackend
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy.openservicemesh.io/v1alpha1","kind":"IngressBackend","metadata":{"annotations":{},"name":"periodicscripttest","namespace":"uop"},"spec":{"backends":[{"name":"periodicscript-v10","port":{"number":8080,"protocol":"https"},"tls":{"skipClientCertValidation":false}}],"sources":[{"kind":"Service","name":"osm-contour-envoy","namespace":"osm-system"},{"kind":"AuthenticatedPrincipal","name":"osm-contour-envoy.osm-system.cluster.local"}]}}
  creationTimestamp: "2022-11-11T16:27:39Z"
  generation: 1
  name: periodicscripttest
  namespace: uop
  resourceVersion: "15379"
  uid: 9f0274c8-cd95-4542-9d7e-9795e0153d47
spec:
  backends:
  - name: periodicscript-v10
    port:
      number: 8080
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    name: osm-contour-envoy
    namespace: osm-system
  - kind: AuthenticatedPrincipal
    name: osm-contour-envoy.osm-system.cluster.local
status:
  currentStatus: error
  reason: endpoints not found for service osm-system/osm-contour-envoy

Kindly ensure only 1 IngressBackend resource is applied for the given backend service. Could you reproduce this issue without the test/Contour configuration as originally seen, and collect the support bundle pertaining to the original issue? As indicated above, the bug-report includes configuration pertaining to Contour. Please remove these debugging configs and provide a bug-report collected immediately after the original issue is reproduced with this backend service.
Thanks

bugreport_Current.zip
hi @shashankram, just now ran the setup with cleanstate and captured the logs. Please find the attached file for details.
FYI... this is development environment which contains various services in the uop namespace. By default as part of the skaffold, we are creating the default ingress endpoints for each service and one of them is "periodicscript-v10'. So, for testing I have created 'periodicscript-test' ingress and corresponding backend to use the OSM certificates as described in the link: https://release-v1-2.docs.openservicemesh.io/docs/demos/ingress_k8s_nginx/#https-ingress-mtls-and-tls.
The ingress is trying to reach the envoy proxy running in the periodicscript pod, and envoy is accepting the connection, but it is closing the connection by saying that there are no filter chain found. below is sample log from the envoy:
[2022-11-21 05:52:13.990][23][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:88] tls inspector: new connection accepted
[2022-11-21 05:52:13.990][23][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:175] tls inspector: recv: 750
[2022-11-21 05:52:13.990][23][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: new connection accepted
[2022-11-21 05:52:13.990][23][trace][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:66] original_dst: set destination to <IP_ADDRESS>:8080
[2022-11-21 05:52:13.990][23][debug][conn_handler] [source/server/active_stream_listener_base.cc:34] closing connection from <IP_ADDRESS>:46690: no matching filter chain found

Updated with additional log messages for ingress controller. Enabled trace for ingress controller.
bugreport_Latest.zip

I looked at the bug report provided in https://github.com/openservicemesh/osm/issues/5247#issuecomment-1321505402 and noticed that the periodicscript-v10 backend does not have an IngressBackend. The only IngressBackend resource available pertains to periodicscript-test, which is not the service we are debugging.
Could you please check if you have correctly deployed the IngressBackend resource?
You do not need to create Envoy filter_chains. OSM controller does that based on the IngressBackend configuration applied.

Hi,
I have recaptured the bugreport after cleanup of old ingress. Now the new ingress periodicscript-v10 has backend for periodicscript service.
the logs are attached
bugreport_3.zip

@shashankram , can you please explain why we get below logs from the envoy...?
[2022-11-21 05:52:13.990][23][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:88] tls inspector: new connection accepted
[2022-11-21 05:52:13.990][23][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:175] tls inspector: recv: 750
[2022-11-21 05:52:13.990][23][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: new connection accepted
[2022-11-21 05:52:13.990][23][trace][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:66] original_dst: set destination to <IP_ADDRESS>:8080
[2022-11-21 05:52:13.990][23][debug][conn_handler] [source/server/active_stream_listener_base.cc:34] closing connection from <IP_ADDRESS>:46690: no matching filter chain found
May be if we understand the rootcause for above messages, we can fix this issue.
One more thing... i have using the self signed certificate for nginx controller for tls.

Based on the annotation  nginx.ingress.kubernetes.io/backend-protocol: GRPC on your ingress resource, it seems like Nginx is not encrypting the connection from the ingress pod -> backend service, but your backend service is configured to accept encrypted traffic based on the following in your IngressBackend config:
  backends:
  - name: periodicscript-v10
    port:
      number: 8080
      protocol: https <-- expects TLS on backend

Please ensure the traffic is being encrypted when Nginx proxies the traffic to the backend. You cannot use TLS passthrough for this. Based on https://kubernetes.github.io/ingress-nginx/examples/grpc/, you may want to set the backend-protocol annotation to HTTPS or GRPCS. Please try changing this annotation to see if it works.

Hi @shashankram , based on your input i have modified the protocol to GRPCS and enabled tls passthrough for ingress controller.
Now envoy returning error : PEER_DID_NOT_RETURN_A_CERTIFICATE
[2022-11-23 11:58:18.252][19][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:88] tls inspector: new connection accepted
[2022-11-23 11:58:18.252][19][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:175] tls inspector: recv: 0
[2022-11-23 11:58:18.252][19][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:175] tls inspector: recv: 203
[2022-11-23 11:58:18.253][19][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:145] tls:onALPN(), ALPN: h2
[2022-11-23 11:58:18.253][19][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:202] tls inspector: done: true
[2022-11-23 11:58:18.253][19][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: new connection accepted
[2022-11-23 11:58:18.253][19][trace][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:66] original_dst: set destination to <IP_ADDRESS>:8080
[2022-11-23 11:58:18.253][19][trace][misc] [source/common/event/scaled_range_timer_manager_impl.cc:60] enableTimer called on 0x454dbf929680 for 3600000ms, min is 3600000ms
[2022-11-23 11:58:18.253][19][debug][conn_handler] [source/server/active_tcp_listener.cc:142] [C41] new connection from <IP_ADDRESS>:56120
[2022-11-23 11:58:18.253][19][trace][main] [source/common/event/dispatcher_impl.cc:249] item added to deferred deletion list (size=1)
[2022-11-23 11:58:18.253][19][trace][main] [source/common/event/dispatcher_impl.cc:125] clearing deferred deletion list (size=1)
[2022-11-23 11:58:18.253][19][trace][connection] [source/common/network/connection_impl.cc:563] [C41] socket event: 3
[2022-11-23 11:58:18.253][19][trace][connection] [source/common/network/connection_impl.cc:674] [C41] write ready
[2022-11-23 11:58:18.254][19][trace][connection] [source/extensions/transport_sockets/tls/ssl_handshaker.cc:52] [C41] ssl error occurred while read: WANT_READ
[2022-11-23 11:58:18.254][19][trace][connection] [source/common/network/connection_impl.cc:603] [C41] read ready. dispatch_buffered_data=0
[2022-11-23 11:58:18.254][19][trace][connection] [source/extensions/transport_sockets/tls/ssl_handshaker.cc:52] [C41] ssl error occurred while read: WANT_READ
[2022-11-23 11:58:18.254][19][trace][connection] [source/common/network/connection_impl.cc:563] [C41] socket event: 3
[2022-11-23 11:58:18.254][19][trace][connection] [source/common/network/connection_impl.cc:674] [C41] write ready
[2022-11-23 11:58:18.254][19][trace][connection] [source/extensions/transport_sockets/tls/ssl_handshaker.cc:52] [C41] ssl error occurred while read: SSL
[2022-11-23 11:58:18.254][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:227] [C41] TLS error: 268435648:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
[2022-11-23 11:58:18.254][19][debug][connection] [source/common/network/connection_impl.cc:250] [C41] closing socket: 0
[2022-11-23 11:58:18.254][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:227] [C41] TLS error: 268435648:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
Please find the attached bugreport.
bugreport_2.zip

Hi @shashankram , based on your input i have modified the protocol to GRPCS and enabled tls passthrough for ingress controller. Now envoy returning error : PEER_DID_NOT_RETURN_A_CERTIFICATE

Could you try HTTPS and see if you see the same?

I have tried same thing with sample https service 'httpbin' given by OSM, and it is working for that. It seems the problem is with the grpc services only.
Could you please let me know if there is anything need to be taken care at the service implementation side to work with OSM? Also, it this OSM tested with GRPC services ...? are there any examples...?

I have tried same thing with sample https service 'httpbin' given by OSM, and it is working for that. It seems the problem is with the grpc services only.
Could you please let me know if there is anything need to be taken care at the service implementation side to work with OSM? Also, it this OSM tested with GRPC services ...? are there any examples...?

If that's the case, this seems to be something to do with how Nginx handles HTTPS vs GRPCS as a backend protocol. From OSM's perspective, only HTTPS based connections are allowed, and gRPC would use HTTP2 so this should ideally work.
You may want to examine if Nginx handles gRPC differently when using TLS.

Hi @shashankram , just now tried a different approach to try a simple service examples given by OSM.
I have followed the example services (bookbuyer, bookstore, booktheif, bookwarehouse) mentioned here https://release-v1-2.docs.openservicemesh.io/docs/getting_started/install_apps/ and deployed all of the services by injecting envoy side car proxy.
By default since all of the namespaces (bookbuyer, bookstore, booktheif, bookwarehouse)  are added to OSM, they are all communicating properly and bookbuyer is able to communicate with bookstore.
Now, I have removed the bookbuyer from the mesh (i.e. kubectl annotate namespace bookbuyer openservicemesh.io/sidecar-injection-) and restarted the bookbuyer.
After this change, the bookbuyer is not able to communicate with bookstore and bookstore is closing the connection with below same error:
[2022-12-02 07:21:44.478][16][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:77] tls inspector: new connection accepted
[2022-12-02 07:21:44.478][16][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:169] tls inspector: recv: 0
[2022-12-02 07:21:44.479][16][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:169] tls inspector: recv: 58
[2022-12-02 07:21:44.479][16][trace][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:191] tls inspector: done: true
[2022-12-02 07:21:44.479][16][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: New connection accepted
[2022-12-02 07:21:44.479][16][debug][conn_handler] [source/server/active_tcp_listener.cc:294] closing connection from <IP_ADDRESS>:50418: no matching filter chain found
This is the same problem i have it with my ingress services... even without using the ingress this problem is exist.
I guess this is nothing to do with nginx... it could be someother problem with osm/configuration.
Could you please let us know what is missing from the configuration side.
I have used the helm charts to install the OSM.
helm install uop-osm osm --repo https://openservicemesh.github.io/osm --version v1.2.2 --namespace osm-system --values override.yaml
And the override.yaml file attached here.
override.zip
And logs are generated using below osm commands:
osm support bug-report --app-namespaces bookstore
--app-deployments bookstore/bookstore
--app-pods bookstore/bookstore-7bb7f9b98c-qplq4
--ingress -o bugreportbookstore.zip
and attached the bugreport here.
bugreportbookstore.zip
Here, my expectation is that the envoy should accept the incoming request from the service which is monitored-by=osm. (FYI.. i have set the label 'openservicemesh.io/monitored-by=osm' for bookbuyer namespace. Still it didnt work.

Also, attached the logger for bookbuyer:
bookbuyer_log.zip

Hi @shashankram  any updates...?
I have created a separate issue to track this issue https://github.com/openservicemesh/osm/issues/5262

Now, I have removed the bookbuyer from the mesh (i.e. kubectl annotate namespace bookbuyer openservicemesh.io/sidecar-injection-) and restarted the bookbuyer.
After this change, the bookbuyer is not able to communicate with bookstore and bookstore is closing the connection with below same error

When you remove a namespace form the mesh, it won't be able to communicate with other meshed apps. You will need to use IngressBackend or IP/Port exclusions to allow non-mesh app to communicate with meshed app.
This is totally unrelated to the Ingress problem you observed with Nginx.
As I mentioned in https://github.com/openservicemesh/osm/issues/5247#issuecomment-1329880251, use HTTPS instead of GRPCS as it works for your use case.

Hi @shashankram , i agree that the bookbuyer issue is not related to this...
Coming to nginx issue, I have tried using the https instead of GRPC in ingress, but it didnt work. I am getting below error when I try to use the nginx endpoint in the browser:
"upstream connect error or disconnect/reset before headers. reset reason: protocol error"
Below is the updated nginx for periodic script service:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: periodicscript-v10
namespace: uop
uid: 5c025ff0-87d3-4ba5-bb25-346ae3008644
resourceVersion: '394519'
generation: 1
creationTimestamp: '2022-12-12T09:33:49Z'
labels:
app: periodicscript-v10
app.kubernetes.io/instance: periodicscript-v10
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: periodicscript
app.kubernetes.io/version: 1.16.0
helm.sh/chart: periodicscript-0.1.0
skaffold.dev/run-id: 93ba5384-4642-4c7b-a729-0f064cfa55bb
version: v1
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
meta.helm.sh/release-name: periodicscript-v10
meta.helm.sh/release-namespace: uop
nginx.ingress.kubernetes.io/backend-protocol: HTTPS  # replaced GRPC with HTTPS as per your inputs.
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_ssl_name "periodicscript-v10.uop.cluster.local";
more_set_headers "X-Frame-Options: DENY";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "Referrer-Policy: strict-origin";
more_set_headers "Permissions-Policy: geolocation=(self)";
more_set_headers "Content-Security-Policy: img-src 'self' data:";
nginx.ingress.kubernetes.io/cors-allow-headers: >-
TenantId,UserId,x-user-agent,authorization,grpc-accept-encoding,grpc-timeout,Access-Control-Allow-Origin,Access-Control-Allow-Credentials,Cache-Control,Access-Control-Allow-Headers,content-type,pragma
nginx.ingress.kubernetes.io/cors-allow-origin: >-
https://aigcdnstorageacct.z22.web.core.windows.net,https://hsprdemoapp.azurewebsites.net
nginx.ingress.kubernetes.io/enable-cors: 'true'
nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
nginx.ingress.kubernetes.io/proxy-body-size: 10m
nginx.ingress.kubernetes.io/proxy-read-timeout: '300'
nginx.ingress.kubernetes.io/proxy-ssl-secret: osm-system/osm-nginx-client-cert
nginx.ingress.kubernetes.io/proxy-ssl-verify: 'on'
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
nginx.ingress.kubernetes.io/use-regex: 'true'
managedFields:
- manager: helm
operation: Update
apiVersion: networking.k8s.io/v1
time: '2022-12-12T09:33:49Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:cert-manager.io/cluster-issuer: {}
f:kubernetes.io/ingress.class: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:nginx.ingress.kubernetes.io/configuration-snippet: {}
f:nginx.ingress.kubernetes.io/cors-allow-headers: {}
f:nginx.ingress.kubernetes.io/cors-allow-origin: {}
f:nginx.ingress.kubernetes.io/enable-cors: {}
f:nginx.ingress.kubernetes.io/force-ssl-redirect: {}
f:nginx.ingress.kubernetes.io/proxy-body-size: {}
f:nginx.ingress.kubernetes.io/proxy-read-timeout: {}
f:nginx.ingress.kubernetes.io/proxy-ssl-secret: {}
f:nginx.ingress.kubernetes.io/proxy-ssl-verify: {}
f:nginx.ingress.kubernetes.io/rewrite-target: {}
f:nginx.ingress.kubernetes.io/ssl-redirect: {}
f:nginx.ingress.kubernetes.io/use-regex: {}
f:labels:
.: {}
f:app: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/managed-by: {}
f:app.kubernetes.io/name: {}
f:app.kubernetes.io/version: {}
f:helm.sh/chart: {}
f:version: {}
f:spec:
f:rules: {}
- manager: skaffold.exe
operation: Update
apiVersion: networking.k8s.io/v1
time: '2022-12-12T09:33:57Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
f:skaffold.dev/run-id: {}
- manager: nginx-ingress-controller
operation: Update
apiVersion: networking.k8s.io/v1
time: '2022-12-12T09:34:10Z'
fieldsType: FieldsV1
fieldsV1:
f:status:
f:loadBalancer:
f:ingress: {}
subresource: status
- manager: node-fetch
operation: Update
apiVersion: networking.k8s.io/v1
time: '2022-12-12T11:29:56Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:nginx.ingress.kubernetes.io/backend-protocol: {}
selfLink: /apis/networking.k8s.io/v1/namespaces/uop/ingresses/periodicscript-v10
status:
loadBalancer:
ingress:
- hostname: localhost
spec:
rules:
- host: uop-cloudservices.<IP_ADDRESS>.nip.io
http:
paths:
- path: /api/uop/periodicscripts/v1(/|$)(.*)
pathType: Prefix
backend:
service:
name: periodicscript-v10
port:
name: grpc

In this case, the nginx it self rejecting the call and the request is not reaching to the envoy proxy in the Pod. It seems this solution is not working... :(
My basic question is that does OSM team tested the GRPC services with ingress backend...? is it working..? Can I get an example of working prototype for GRPC services with ingress backend....?
Its been more than a month I have been trying to resolve this issue, but till today I didnt get any concrete solution to fix this issue. It will be impacting our general release which is few months away from now.
Please let us know the next steps... whether should we continue use OSM for GRPC services or switch to Istio which is working smoothly without any issue....

I have tried same thing with sample https service 'httpbin' given by OSM, and it is working for that. It seems the problem is with the grpc services only.

Based on your earlier comment in https://github.com/openservicemesh/osm/issues/5247#issuecomment-1328658097, seems like it worked for you?
Please provide a standlone repro with a simple gRPC service. There are many sample gRPC services available (e.g grpcbin) that you can test this out with.

@shashankram how about we excluding nginx ingress controller from the picture? Just focus on gRPC traffic handling in an OSM sidecared pod.
According to OSM doc, the mesh detects and allows gRPC traffic into the pod if "appProtocol" is set to gRPC in svc. I did that:
spec:
  ports:
    - protocol: TCP
      appProtocol: gRPC
      port: 8080
      targetPort: 8080

Then I got connection refused failure at my gRPC client:
# ./client grpc-ping-srv.default:8080
2022/12/15 06:16:01 Trying to dial: grpc-ping-srv.default:8080
2022/12/15 06:16:01 Trying to new
2022/12/15 06:16:01 Trying to say hi
2022/12/15 06:16:01 Error when calling SayHello: rpc error: code = Unavailable desc = connection closed before server preface received

In envoy logs of the target pod:
2022-12-15T06:16:01Z {"time_to_first_byte":null,"path":null,"bytes_received":0,"response_flags":"NR","authority":null,"response_code_details":"filter_chain_not_found","upstream_host":null,"method":null,"bytes_sent":0,"upstream_service_time":null,"upstream_cluster":null,"request_id":null,"user_agent":null,"requested_server_name":null,"x_forwarded_for":null,"start_time":"2022-12-15T06:16:01.168Z","protocol":null,"response_code":0,"duration":0}

Note that I do have permissive traffic mode on:
apiVersion: config.openservicemesh.io/v1alpha2
kind: MeshConfig
metadata:
  name: osm-mesh-config
spec:
  traffic:
    enableEgress: true
    enablePermissiveTrafficPolicyMode: true

Based on the above findings, I do not think OSM is gRPC ready.

Thank god... at list one more person is having the similar issue with GRPC service.
I have extended my analysis and run osm verify ingress CLI command below is the output:
i. First I tried with IngressBackend protocol as 'https' like below:
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
name: periodicscript-v10
namespace: uop
spec:
backends:

name: periodicscript-v10
port:
number: 8080
protocol: https # Note that protocol is https....
tls:
skipClientCertValidation: false
sources:
kind: Service
namespace: ingress-nginx
name: ingress-nginx-controller
kind: AuthenticatedPrincipal
name: ingress-nginx.ingress-nginx.cluster.local

The CLI command result:
PS C:\Users\ramachandraiah.pandu\Downloads\osm-v1.2.2-windows-amd64\windows-amd64> .\osm.exe verify ingress --from-service ingress-nginx/ingress-nginx-controller --to-pod uop/periodicscript-v10-7668554b4c-tmrd9 --to-service periodicscript-v10 --ingress-backend periodicscript-v10 --to-port 8080
[+] Context: Verify if service "ingress-nginx/ingress-nginx-controller" can access pod "uop/periodicscript-v10-7668554b4c-tmrd9" for app protocol "http" via ingress
Status: ←[31mFailure←[0m
Reason: A verification step failed
Suggestion: Please follow the suggestions listed in the failed steps below to resolve the issue
[++] Context: Verify if namespace "ingress-nginx" is monitored
Status: ←[32mSuccess←[0m
[++] Context: Verify if namespace "uop" is monitored
Status: ←[32mSuccess←[0m
[++] Context: Verify Envoy sidecar on pod "ingress-nginx/ingress-nginx-controller-7d9564c96c-mbgb5"
Status: ←[32mSuccess←[0m
[++] Context: Verify Envoy sidecar on pod "uop/periodicscript-v10-7668554b4c-tmrd9"
Status: ←[32mSuccess←[0m
[++] Context: Verify IngressBackend "uop/periodicscript-v10"
Status: ←[31mFailure←[0m
Reason: No backend matching port 8080 and protocol http found in IngressBackend "uop/periodicscript-v10"
Suggestion: Add a backend matching port 8080 and protocol http found in IngressBackend "uop/periodicscript-v10"
[++] Context: Verify Envoy config for traffic:
source service: ingress-nginx/ingress-nginx-controller
destination pod: uop/periodicscript-v10-7668554b4c-tmrd9
destination service: uop/periodicscript-v10
destination protocol: http
Status: ←[31mFailure←[0m
[+++] Context: Verify Envoy config on destination
Status: ←[31mFailure←[0m
Reason: Did not find matching inbound filter chain for service "uop/periodicscript-v10": unsupported protocol TCP

Then I have changed the ingressbackend port to 'http'. In this case CLI command result is different. The 'verify ingressbackend' is success. But the envoy config for traffic/destination is failling.
Below is the result:
PS C:\Users\ramachandraiah.pandu\Downloads\osm-v1.2.2-windows-amd64\windows-amd64> .\osm.exe verify ingress --from-service ingress-nginx/ingress-nginx-controller --to-pod uop/periodicscript-v10-7668554b4c-tmrd9 --to-service periodicscript-v10 --ingress-backend periodicscript-v10 --to-port 8080

[+] Context: Verify if service "ingress-nginx/ingress-nginx-controller" can access pod "uop/periodicscript-v10-7668554b4c-tmrd9" for app protocol "http" via ingress
Status: ←[31mFailure←[0m
Reason: A verification step failed
Suggestion: Please follow the suggestions listed in the failed steps below to resolve the issue
[++] Context: Verify if namespace "ingress-nginx" is monitored
Status: ←[32mSuccess←[0m
[++] Context: Verify if namespace "uop" is monitored
Status: ←[32mSuccess←[0m
[++] Context: Verify Envoy sidecar on pod "ingress-nginx/ingress-nginx-controller-7d9564c96c-mbgb5"
Status: ←[32mSuccess←[0m
[++] Context: Verify Envoy sidecar on pod "uop/periodicscript-v10-7668554b4c-tmrd9"
Status: ←[32mSuccess←[0m
[++] Context: Verify IngressBackend "uop/periodicscript-v10"
Status: ←[32mSuccess←[0m
[++] Context: Verify Envoy config for traffic:
source service: ingress-nginx/ingress-nginx-controller
destination pod: uop/periodicscript-v10-7668554b4c-tmrd9
destination service: uop/periodicscript-v10
destination protocol: http
Status: ←[31mFailure←[0m
[+++] Context: Verify Envoy config on destination
Status: ←[31mFailure←[0m
Reason: Did not find matching inbound filter chain for service "uop/periodicscript-v10": unsupported protocol TCP
It seems the envoy proxy doesnt have matching inbound filter 'TCP' or something else....
I hope this additional information provides some insights of OSM support for grpc.

Saw a relevant error,
$ kubectl edit ingressbackend/grpcbin
error: ingressbackends.policy.openservicemesh.io "ingressbackend-grpcbin-https" could not be patched: admission webhook "osm-validator.k8s.io" denied the request: Expected 'port.protocol' to be 'http' or 'https', got: grpc

Also, even if tried to used https as port.protocol the IngressBackend still unable to gain the "committed" state.

Hi All,
Finally, I am able to work with below configuration:
i. Update the appProtocol: grpc in service.yaml
ii. Create ingressbackend with port.Protocol = http.
After making above changes, I am able to communicate with services using my ingress backend.
But, still https ingress backend (port.Protocol=https) is not working. This needs to be fixed.

Hi all,
Jumping in on this, it seems there's a lot of history above. Can you summarize what the current issues are and provide a simple repro? This can be just the yaml files, using ie: grpc-bin, or a public echo server image.
From there we can see what is wrong and try to recommend a fix, or see if there's an actual bug in OSM.

Just give a background:
I have a grpc service which is communicating with other services in the same namespace. I want to integrate with OSM... after lot of struggle finally I could able to work with below configuration:
i. Updated the service.yaml file with appProtocol: grpc and ingressbackend.yaml with Protocol: http. With this it is working.
But, as per the documentation given @ https://release-v0-11.docs.openservicemesh.io/docs/demos/ingress_k8s_nginx/, we can also enable ingressbackend with Protocol:https.
I have followed the test suite "https://github.com/openservicemesh/osm/blob/main/tests/e2e/e2e_grpc_secure_origination_test.go" to enable https communication between the nginx and service. So, I have set the appProtocol: tcp and ingressbackend: https. But this is not working. The envoy proxy is closing the connection.
I have used the osm cli to verify the communication between the ingress and service, the command returned below result:
The CLI command result:
PS C:\Users\ramachandraiah.pandu\Downloads\osm-v1.2.2-windows-amd64\windows-amd64> .\osm.exe verify ingress --from-service ingress-nginx/ingress-nginx-controller --to-pod uop/periodicscript-v10-7668554b4c-tmrd9 --to-service periodicscript-v10 --ingress-backend periodicscript-v10 --to-port 8080
[+] Context: Verify if service "ingress-nginx/ingress-nginx-controller" can access pod "uop/periodicscript-v10-7668554b4c-tmrd9" for app protocol "http" via ingress
Status: ←[31mFailure←[0m
Reason: A verification step failed
Suggestion: Please follow the suggestions listed in the failed steps below to resolve the issue
[++] Context: Verify if namespace "ingress-nginx" is monitored
Status: ←[32mSuccess←[0m
[++] Context: Verify if namespace "uop" is monitored
Status: ←[32mSuccess←[0m
[++] Context: Verify Envoy sidecar on pod "ingress-nginx/ingress-nginx-controller-7d9564c96c-mbgb5"
Status: ←[32mSuccess←[0m
[++] Context: Verify Envoy sidecar on pod "uop/periodicscript-v10-7668554b4c-tmrd9"
Status: ←[32mSuccess←[0m
[++] Context: Verify IngressBackend "uop/periodicscript-v10"
Status: ←[31mFailure←[0m
Reason: No backend matching port 8080 and protocol http found in IngressBackend "uop/periodicscript-v10"
Suggestion: Add a backend matching port 8080 and protocol http found in IngressBackend "uop/periodicscript-v10"
[++] Context: Verify Envoy config for traffic:
source service: ingress-nginx/ingress-nginx-controller
destination pod: uop/periodicscript-v10-7668554b4c-tmrd9
destination service: uop/periodicscript-v10
destination protocol: http
Status: ←[31mFailure←[0m
[+++] Context: Verify Envoy config on destination
Status: ←[31mFailure←[0m
Reason: Did not find matching inbound filter chain for service "uop/periodicscript-v10": unsupported protocol TCP

thanks for the context! would you be able to provide the yaml manifests for a minimum reproducible setup?

i guess, I have provided a lot of information in this ticket and still you are asking me for additional information. I dont understand why....?
Simple thing: the communication is working with appProtocol: grpc and ingress backedn protocol as 'http'. This is insecure version of ingress.
Now, I wanted to try secure version of ingress by updating the 'appProtocol:tcp; and ingress backedn as 'https' which is not working.
As part of the OSM development is this scenario tested with ingress...? if yes, could you please share the right configuration?
Initially, I have provided all of the configuration including the service & ingress configuration. But I am not sure shashankram gone through this configuration. Otherwise the simple configuration changes like 'appProtocol:grpc' and backend protocol as 'http' is working.

Hey @rcpandula, I understand your frustration. There's a lot of context on this bug, and it looks like there's been some changes to what is being tested? At one point focus was moved to the bookstore demo, then moved away, then once on nginx, then removed.
There's also a lot of configuration in OSM that can affect the envoy sidecar and it's configuration. Things like service, ingressbackend, meshconfig, whether pods are statefulsets or not, and more. Having a reproducible environment allows us to make sure we are working on the exact problem you are describing, with the exact configuration, and allows us new engineers to spin up more quickly (ie: @shashankram is no longer a maintainer).
A minimum reproducible environment is a typical ask in an open source setting. We can still investigate without one, but it may fall a bit lower on a priority stack if user's opt to not provide one. I hope that context helps!

a quick question will help me to solve this issue.
For plain http communication, we are setting the appProtocol as grpc and ingressbackend:Protocol as http. so, what is the combination for enabling the https communication from ingress to actual service.
In case of http, the traffic from ingress controller to service is not encrypted (as it is told @ https://release-v1-2.docs.openservicemesh.io/docs/demos/ingress_k8s_nginx/), so how to enable https for grpc services...?
Please provide these responses it will really helpful.

I dug into this more today, this looks to be an "as intended" feature of nginx https://github.com/kubernetes/ingress-nginx/issues/3897
That is, nginx ingress opens 2 ports, 80 and 443. 80 only supports HTTP 1.1 while 443 supports HTTP 2. grpc occurs over HTTP 2, so you'll need to use TLS from your client to nginx.
With all of that in mind, I got this working, but if you want to have TLS between nginx and your grpc service, you'll need to use TLS between client and nginx as well. The other option is no TLS between anything.
To get TLS working, you'll want to set nginx.ingress.kubernetes.io/backend-protocol: "GRPCS" as an annotation on your ingress object.  and use https as the protocol on the ingressbackend.
You'll also need to provide a TLS block to ensure TLS is terminated at nginx.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grpc-echo-ingress
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPCS"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_ssl_name "grpc-echo.default.cluster.local";
      grpc_read_timeout 3600s;
    # proxy_ssl_name for a service is of the form <service-account>.<namespace>.cluster.local
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "osm-system/osm-nginx-client-cert"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
spec:
  tls:
    - hosts:
      - your.host.com
      secretName: ssl-term
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grpc-echo
            port:
              number: 9000
---
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: grpc-echo
spec:
  backends:
  - name: grpc-echo
    port:
      number: 9000
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: nginx
    name: ingress-nginx-controller
  - kind: AuthenticatedPrincipal
    name: ingress-nginx.nginx.cluster.local


getting connecttion refused when using playwright-lighthouse with url

getting connecttion refused when using playwright-lighthouse with ur in playAudit({url: 'someAuthenticatedurl'})
Error: connect ECONNREFUSED <IP_ADDRESS>:53938

Make sure you are setting a debugging port in playwright config under defineConfig:
    use: {
        /* Start browser in debug port mode; allows lighthouse access to the browser port */
        launchOptions: {
            args: ['--remote-debugging-port=9222'],
        },
    },


Addition of Option for Weight Average Calculation in Heatmap

Hello OpenLayers team,
I hope this message finds you well!
I am using the Heatmap component of OpenLayers in my project and I have observed that the current functionality, which includes options for blur, radius, and weight, works very well for summable data such as population or accident counts. However, I am dealing with a specific case where summing weights is not appropriate, such as when visualizing prices by region (e.g., fuel prices by neighborhood).
For data like prices, calculating the average of the values is crucial for accurate visualization. Currently, the heatmap sums the weights when coordinates are close to each other, but in my case, having an option to calculate the average of these weights would be far more useful.
I would like to suggest considering the addition of an option to compute the average of weights in the heatmap. This would provide a more accurate and meaningful representation of price data and similar metrics.
I also wanted to point out that this topic has been previously raised in issue #11819 (Heatmap layer intersection between points), but it was closed due to inactivity. I believe this enhancement would greatly benefit the community, and I hope it can be revisited.
Thank you very much for the excellent work on OpenLayers, which has been instrumental not only for my project but for many others around the world. 😃
I look forward to your response and am available for any additional information if needed.

To be honest, I don't know how a heat map of averages would work. The idea behind the heatmap is to draw a point with a color according to its weight (giving it the red-yellow heat effect), and when all points are drawn, blur the resulting image.
For data like prices, given my cartographic knowledge, I would create a choropleth or raster map that aggregates points with price attributes into the respectiv regional units or raster cells, by average.

Hi @ahocevar, thank you so much for your response and for your time! I know that the OpenLayers project, especially now with version 10, requires a lot of effort, so I really appreciate your attention.
I just wanted to gently insist on the use of averages in the heatmap. Although I mentioned prices as an example, the average also makes a big difference for other data types like temperature, air quality, and pollution levels.
To illustrate, I’m sending a figure showing two circles representing temperature measurements: 28°C (#B8FFF7) and 42°C (#94FFB7). With the blur turned off to make it clearer, the overlap of the circles results in 70°C (#9AFF6B), which doesn’t make much sense for temperature. Similarly, for air quality data or pollution indices, the average provides a more accurate view of the environmental impact.

It would be great to have the option to calculate the average of weights in the heatmap. This would help provide a more precise and useful visualization for these types of data.
I’m testing the latest version of OpenLayers and am really pleased with the results. Thanks again for your consideration and for the excellent work on the tool!

Sorry to insist, but what I tried to explain above is that the heatmap is not something that's calculated, but the result of drawing pixels with a fixed color scale for values between 0  and 1, using a "source-over" composite operation, and then applying a blur to the final result.
What you're asking for is drawing pixels with a user defined color scale for arbitrary values, using an "overlay" composite operation, and then apply a blur to the final result. This can be done with a vector layer, prerender and postrender functions, and a blur css filter with a className on the layer.
See https://codesandbox.io/p/sandbox/vector-layer-forked-mvw2ls for an example.

The result of overlay would be affected by the rendering order of the features.  Whatever operation is used it is unlikely to give meaningful results if 3 or more features overlap (e.g. https://openlayers.org/en/v5.3.0/examples/blend-modes.html).

data types like temperature, air quality, and pollution levels

These are typically represented as a raster using an Inverse Distance Weight (IDW) interpolation algorithm which can handle multiple randomly distributed sample points.  None of the major mapping clients support that directly but all have third-party extensions available, e.g. https://stackblitz.com/edit/js-z4ota1?file=package.json,index.html,index.js  Sometimes called an "interpolated heatmap" but it is not a true heatmap as each pixel on the output must be calculated.

I had the same problem and by implementing such algorithm (e.g. idw) can slow down the entire application when you have a lot of data.
I came to this compromise: implement the Heatmap layer only by using deck.gl which already implement the MEAN aggregation feature. Below an example of how this can be done:
const _9ClassYlOrRd: any = [[255, 255, 204], [255, 237, 160], [254, 217, 118], [254, 178, 76], [253, 141, 60], [252, 78, 42], [227, 26, 28], [189, 0, 38], [128, 0, 38]];
function calculateWeight(n: number, minRange: number, maxRange: number): number {
  const rangeLength = maxRange - minRange;
  let result = (n - minRange) / rangeLength;
  result = Math.max(Math.min(result, maxClamp), minClamp);
  return result;
}

export class HeatmapLayer extends BaseLayer<Layer> {

  private urlBuilder = this.urlFn(this.layerName);
  private deckView = new Deck({
    initialViewState: { longitude: 0, latitude: 0, zoom: 1 },
    controller: false,
    parent: document.getElementById('mapElement') as HTMLDivElement,
  });

  constructor(options: LayerOptions, private progressSource: ProgressSourceService) {
    super(options)
  }

  private heatmap(data: string) {
    return new DeckHeatmap({
      id: 'HeatmapLayer',
      aggregation: 'MEAN',
      data,
      intensity: .5,
      colorRange: _9ClassYlOrRd,
      getPosition: d => d.coordinates,
      getWeight: d => d.weight,
      dataTransform: this.transformData.bind(this),
      radiusPixels: 10,
      fetch: (url, ctx) => {
        this.progressSource.emitStart();
        return fetch(url, ctx).then(r => r.json())
      },
      onDataLoad: () => this.progressSource.emitEnd()
    });
  }

  protected override initLayer() {
    return new Layer({ render: this.renderFunc.bind(this) });
  }

  override changeFeature(name: string): void {
    this.options.additionalParams = `propertyName=the_geom,${name}`;
    this.options.feature = name;
    super.changeFeature(this.options.additionalParams);
  }

  private renderFunc(frameState: FrameState) {
   // chatgpt code - start
    const [width, height] = frameState.size;
    const viewState = frameState.viewState;
    const [longitude, latitude] = toLonLat(viewState.center, viewState.projection);
    const zoom = viewState.zoom - 1;
    const bearing = (-viewState.rotation * 180) / Math.PI;
    const deckViewState = { bearing, longitude, latitude, zoom };
   // chatgpt code - end
    const builtUrl = this.urlBuilder(frameState.extent!);
    const layer = this.heatmap(builtUrl);
    this.deckView.setProps({ width, height, viewState: deckViewState, layers: [layer] });
    this.deckView.redraw();
    return this.deckView.getCanvas()!;
  }

  private transformData(data: any, previousData: any) {
    const features: any[] = data.features;
    const newData = features.map(f => {
      const value = f.properties[this.options.feature];
      const calculated = calculateWeight(value, minRange, maxRange);
      const coords = f.geometry?.coordinates;
      return ({ coordinates: coords, weight: calculated })
    })
    if (previousData !== undefined) {
      newData.push(...previousData)
    }
    return newData as any;
  }
}

In my case, I'm using a wfs service to load data. Hope this can help.


SDK version 3.0 change to user attributes value NSObject makes swift API awkward

From release notes:

Previously, notification listeners were only given string-valued user attributes because only strings could be passed into various method calls. That is no longer the case. The ActivateListener and TrackListener interfaces now receive user attributes as NSDictionary<NSString *, NSObject *> * _Nullable instead of NSDictionary<NSString *,NSString *> * _Nonnull.

This makes the Swift API awkward because there in Swift there is no implicit conversion to NSObject.
Eg, we get a lot of errors:

Cannot convert return expression of type '[String : String]' to return type '[String : NSObject]?'
Cannot convert return expression of type '[String : Any]' to return type '[String : NSObject]?'

The Objective-C header should instead declare the dictionary as  NSDictionary<NSString *, id> * which gets bridged to [String : Any].
Alternatively, you could put a NS_REFINED_FOR_SWIFT on those API methods and provide a type safe Swift API.

+1

Thanks for reporting this issue @bobergj . We will take a look and get back to you.

We are looking at a patch release using NSDictionary<NSString*,id>*.   We are also updating event tags and all attribute calls where this would be an issue.  Again, thanks for reporting!

@bobergj @Nonnus We just released v3.0.2 which removes the need for the attribute type casting. Thanks for your help!

@bobergj @Nonnus let us know if you continue to run into the issue in 3.0.2
Going to mark this issue as closed. Please re-open in case.
Thanks for reporting this !!


Move remaining tests from libgdf/tests to libgdf/src/tests

Most tests were moved to libgdf/src/tests in https://gitlab-master.nvidia.com/RAPIDS/libgdf/commit/1850b5f1238cc1d1e09d2fac7d03c15ca64badfd but there are still a few in libgdf/tests that need to be moved.

@yashv28 can you look into this one?

Done


[AppConfig] Include the version updates for external libraries.

Preview release PR don't include the external library version updates, it failed the CI check pipeline, https://dev.azure.com/azure-sdk/internal/_build/results?buildId=687930&view=logs&j=36df4387-3551-57fa-098c-b43a23930261&t=1c264aeb-6cc0-57a9-5463-c48be5861afd
This is a follow up PR for https://github.com/Azure/azure-sdk-for-java/pull/18620

/check-enforcer override

/check-enforcer override


NM - CLIA Report Dashboard - no longer receiving

Describe the bug
New Mexico has not received a CLIA report since May 11th, 2022.
From: Nweke, Pearl, DOH<EMAIL_ADDRESS>Sent: Tuesday, June 21, 2022 5:50 PM
To<EMAIL_ADDRESS>Cc: Dinwiddie, Jordyn, DOH<EMAIL_ADDRESS>Smith, Carmela, DOH<EMAIL_ADDRESS>Report Stream (CDC<EMAIL_ADDRESS>Ligh, Ming (CDC/OD/OCS<EMAIL_ADDRESS>Doyle, Maggie, DOH<EMAIL_ADDRESS>Subject: New Mexico CLIA Report Dashboard
Good afternoon Simple Report,
New Mexico has not received a CLIA report since May 11th, 2022. Please, can you send us the latest report? We will like to receive the report twice a week on Mondays and Wednesdays.
Let us know if you have any questions.
Thank you
Pearl
Expected behavior
Please, can you send us the latest report? We will like to receive the report twice a week on Mondays and Wednesdays.
Additional context
After looking in the SLLT folder for New Mexico. It appears that the CLIA reports have been sent to Pearl. Not sure why she did not receive it.

@sliu1000 I just reconfigured their report distribution. Lets check in with them and see if that takes care of their issue.

@Adrian-Brewster Hi Adrian,  I have contacted Pearl and let her know.


Missing license information

Which license is this the Android logger project under?

While one is not specified I would imagine that if we were to add one in it would be the MIT license.

It would be really good if everything was explicitly licensed as MIT so
that there are no doubts about the usability of this library in commercial
products.
2015-04-07 12:05 GMT+02:00 Stephen Hynes<EMAIL_ADDRESS>
While one is not specified I would imagine that if we were to add one in
it would be the MIT license.
—
Reply to this email directly or view it on GitHub
https://github.com/logentries/le_android/issues/19#issuecomment-90494459
.

Agreed - we need a license for this code.  Many organisations would not include this code without explicit licensing.

I'll add license in a sec...

Done! Thanks @hirro @snodnipper


Remove FFmpeg normalization causing "quiet" songs.

Title
Audio downloaded with spotdl is relatively quiet.
Description
Audio normalization flags in the FFmpeg command reduces actual song loudness by almost 50%
Motivation and Context
dr.boaf on discord originally voiced the issue and I too took notice of it. Removing the normalization flags causes the converted audio to match closer to the actual song. The normalization causes the song to be very dull and can be very apparent in certain conditions. I believe @MikhailZex originally introduced the flags. Could you expand on the usage and purpose?
Testing / Screenshots (if appropriate)
Subject: Tristam - Over the Edge

The absence of apad_dir also makes the song the exact same length
Types of Changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project
[ ] My change requires a change to the documentation
[ ] I have updated the documentation accordingly
[x] I have read the CONTRIBUTING document
[x] I have read the  CORE VALUES document
[ ] I have added tests to cover my changes
[x] All new and existing tests passed

Just fixed a flag that increased bitrate. See https://trac.ffmpeg.org/wiki/Encode/MP3
before/after

The dullness is due to the dynaudnorm flag that causes local amplitude modulation instead of global amplitude modulation. This ensures consistent loudness with almost no variation - variation that's supposed to be an integral part of a song.
It has been removed multiple times in the past and then been reintroduced again and again. Kind of gave up on getting it out.
Try conversion of the song with only loudnorm, if you still feel the audio is dull, go ahead with this PR as is, else remove all flags except loudnorm.
You can checkout my personal modified spotdl copy with alternate flags here.

Please pull changes from upstream/dev

My question is why is the audio modification in there at all. I feel like most people would want the unmodified audio.

Instrumental music on YouTube has wayyy to much loudness variation. Without normalisation, your have to change the volume on a song to song basis. This effect is far less pronounced for mainstream music.
P.S. It's mostly for that reason that Spotify itself implemented loudness normalisation in songs by default.

That makes sense. I will mess with the normalization to see if I could improve it. For now, though, I feel it would be better to have inconsistently varying songs rather than a quieted song (in place over an already compressed mp3 file). Most older songs were naturally quieter.

I feel like the dynaudnorm should be left out altogether since it applies normalization dynamically over windowed portions of the file and changed characteristics of the song. For example, some songs intend areas to be quiet while others to be loud.

I feel like the dynaudnorm should be left out altogether since it applies normalization dynamically over windowed portions of the file and changed characteristics of the song. This is not desired since some songs intend areas to be quiet while others to be loud.

Exactly what I said:

The dullness is due to the dynaudnorm flag that causes local amplitude modulation instead of global amplitude modulation. This ensures consistent loudness with almost no variation - variation that's supposed to be an integral part of a song.
It has been removed multiple times in the past and then been reintroduced again and again. Kind of gave up on getting it out.
Try conversion of the song with only loudnorm, if you still feel the audio is dull, go ahead with this PR as is, else remove all flags except loudnorm.
You can checkout my personal modified spotdl copy with alternate flags here.

From my set of experimentation, one other thing of note is setting a relatively high level for loudnorm (eg. loudnorm=-3) will cause "pumping" - rhythmic loudness-softness loops.


Bozja maps

Can you please add the bosses map for all Bozja's maps? Some noticeable ones like the Red Chocobo and the Robots at the 3rd map

Plan to do it eventually, not very soon due to lack of time though.

Outdated. Module are added as they are created.


Mind if I add babel and update some syntax?

Goals

try to make the repo show up as js rather than html
update some very minor things in the readme

Awesome repo! 🙌 —which is why I figured I might help (but don't want to bother).
If it's cool, I'll submit a pr.

Add babel? For what? What syntax are you referring to?

It would just be updating the repo to use es6 (not needed at all). 😊

Also, with template literals, it could look pretty clean to render tests as mainly JavaScript so that the repo would show up as a JavaScript Repository (see above).
As far as the readme:

date update
yarn install

~Thanks

Thanks for the offer, but there is no pressing reason to change to es6. If we were refactoring already and there was something in es6 that provided an advantage for something we needed to accomplish then I'd be in favor of it.  I do appreciate it.


Math (e.g. Log1p()) functions run in ScriptRunTimeError on floating point exceptions

Describe the issue you are experiencing
The following code causes a ScriptRuntimeError if executed in the Skript testen dialog:
WriteLine("START");

! works
real a = -1.0;
real b = -2.0;
WriteLine(a.Log1p().ToString(15)); ! expected: -inf
WriteLine(b.Log1p().ToString(15)); ! expected: -nan

! runs into scriptruntimeerror
real aa = a.Log1p();
real bb = b.Log1p();
WriteLine(aa.ToString(15));
WriteLine(bb.ToString(15));

WriteLine("DONE");

Describe the behavior you expected
This above ReGa-script should not run into a ScriptRunTimeError if Log1p(-1.0) or similar floating point exceptions are happening within Log1p(). Interesting enough, executing Log1p(-1.0).ToString() seems to work because the runtime error will only be triggered when trying to assign the return value from Log1p() which was running in the floating point exception.
Steps to reproduce the issue

Execute above code
Monitor /var/log/messages logfile for ScriptRunTimeErrors
comment out last section and run again and note that DONE is then correct output

What is the version this bug report is based on?
<IP_ADDRESS>30225
Which base platform are you running?
ova (Open Virtual Infrastructure)
Which HomeMatic/homematicIP radio module are you using?
RPI-RF-MOD
Anything in the logs that might be useful for us?
Apr  3 20:30:59 homematic-raspi2 local0.err ReGaHss: ERROR: ScriptRuntimeError: WriteLine("START");  ! works real a = -1.0; real b = -2.0; WriteLine(a.Log1p().ToString(15)); ! expected: -inf WriteLine(b.Log1p().ToString(15)); ! expected: -nan  ! runs into scriptruntimeerror real aa = a.Log1p(); real bb = b.Log1p(); WriteLine(aa.ToString(15)); WriteLine(bb.ToString(15));  WriteLine("DONE"); [ExecError():iseESPexec.cpp:12882]

Additional information
No response

After somewhat more analyses on that matter other math functions seem to be also affected by this issue. The following ReGa script reveals that matter:
WriteLine("START");

! works
real a = -1.0;
real b = -2.0;
WriteLine(a.Log1p().ToString(15)); ! expected: -inf
WriteLine(b.Log1p().ToString(15)); ! expected: -nan

! Log1p()
real inf = a.Log1p(); ! expected: -inf
real nan = b.Log1p(); ! expected: -nan
WriteLine(inf);
WriteLine(nan);

! Mod()
WriteLine(inf.Mod(2)); ! expected: -nan
WriteLine(nan.Mod(2)); ! expected: -nan
a = 2.0;
WriteLine(a.Mod(0)); ! expected: -nan

! Exp()
WriteLine(inf.Exp()); ! expected: 0.000000
WriteLine(nan.Exp()); ! expected: -nan
a = 1000000.0;
WriteLine(a.Exp()); ! expected: inf (overflow)

! Exp2()
WriteLine(inf.Exp2()); ! expected: 0.000000
WriteLine(nan.Exp2()); ! expected: -nan
a = 1000000.0;
WriteLine(a.Exp2()); ! expected: inf (overflow)

! Exp10()
WriteLine(inf.Exp10()); ! expected: 0.000000
WriteLine(nan.Exp10()); ! expected: -nan
a = 1000000.0;
WriteLine(a.Exp10()); ! expected: inf (overflow)

! Expm1()
WriteLine(inf.Expm1()); ! expected: -1.000000
WriteLine(nan.Expm1()); ! expected: -nan
a = 1000000.0;
WriteLine(a.Expm1()); ! expected: inf (overflow)

! Log()
WriteLine(inf.Log()); ! expected: -nan
WriteLine(nan.Log()); ! expected: -nan
a = 0.0;
WriteLine(a.Log()); ! expected: -inf

! Log2()
WriteLine(inf.Log2()); ! expected: -nan
WriteLine(nan.Log2()); ! expected: -nan
a = 0.0;
WriteLine(a.Log2()); ! expected: -inf

! Log10()
WriteLine(inf.Log10()); ! expected: nan
WriteLine(nan.Log10()); ! expected: -nan
a = 0.0;
WriteLine(a.Log10()); ! expected: -inf

! Log1p()
WriteLine(inf.Log1p()); ! expected: -nan
WriteLine(nan.Log1p()); ! expected: -nan
a = -1.0;
WriteLine(a.Log1p()); ! expected: -inf

! Sqrt()
WriteLine(inf.Sqrt()); ! expected: -nan
WriteLine(nan.Sqrt()); ! expected: -nan
a = -1.0;
WriteLine(a.Sqrt()); ! expected: -nan

! Pow()
WriteLine(inf.Pow(1)); ! expected: -inf
WriteLine(nan.Pow(1)); ! expected: -nan
a = 0.0;
WriteLine(a.Pow(-1)); ! expected: inf

! Sin()
WriteLine(inf.Sin()); ! expected: -nan
WriteLine(nan.Sin()); ! expected: -nan

! Cos()
WriteLine(inf.Cos()); ! expected: -nan
WriteLine(nan.Cos()); ! expected: -nan

! Tan()
WriteLine(inf.Tan()); ! expected: -nan
WriteLine(nan.Tan()); ! expected: -nan

! Asin()
WriteLine(inf.Asin()); ! expected: nan
WriteLine(nan.Asin()); ! expected: -nan
a = 10.0;
WriteLine(a.Asin()); ! expected: nan

! Acos()
WriteLine(inf.Acos()); ! expected: nan
WriteLine(nan.Acos()); ! expected: nan
a = 10.0;
WriteLine(a.Acos()); ! expected: nan

! Atan()
WriteLine(inf.Atan()); ! expected: -1.570796 (-pi/2)
WriteLine(nan.Atan()); ! expected: -nan

! Sinh()
WriteLine(inf.Sinh()); ! expected: -inf
WriteLine(nan.Sinh()); ! expected: -nan
a = 1000000.0;
WriteLine(a.Sinh()); ! expected: inf (overflow)

! Cosh()
WriteLine(inf.Cosh()); ! expected: inf
WriteLine(nan.Cosh()); ! expected: -nan
a = 1000000.0;
WriteLine(a.Cosh()); ! expected: inf (overflow)

! Tanh()
WriteLine(inf.Tanh()); ! expected: -1.000000
WriteLine(nan.Tanh()); ! expected: -nan

! Asinh()
WriteLine(inf.Asinh()); ! expected: -inf
WriteLine(nan.Asinh()); ! expected: -nan

! Acosh()
WriteLine(inf.Acosh()); ! expected: -nan
WriteLine(nan.Acosh()); ! expected: -nan
a = -1.0;
WriteLine(a.Acosh()); ! expected: -nan

!


cloudscribe add-in for Visual Studio for Mac

I didn't see any cloudscribe Templates add-in when searching in Visual Studio for Mac 2019.  Is that coming soon?  Need help with that?

https://docs.microsoft.com/en-us/visualstudio/mac/extending-visual-studio-mac-walkthrough?view=vsmac-2019
http://addins.monodevelop.com/

@trendoid building a plugin for VS Mac is not a priority for me. We have a plugin for VS2017 and VS2019, but it has to be installed from the marketplace, it doesn't just show up in VS automagically. Hopefully someday Microsoft will unify VS and VS Mac so that the same plugins work in both.
We also have a template that works from the command line with dotnet new command as discussed here which can be used on any .net core supported platform, so that would be what I would use on Mac or Linux.

@joeaudette Thanks.  I'll take a look at that CLI command.


[choreo] Fix heading on non-pose waypoints

This works on well-formed choreo paths. It adjusts the headings of translation and empty waypoints based on constraints and surrounding pose waypoints.
The next step is to provide feedback to the user about which waypoint(s) have conflicting constraints regarding heading.

this is broken after merging in main.

broken again after merging in main.


404

Why
As a developer and user, I want consistent and specified return statements for each function
What
Added a 404 test that will return None, empty list, and empty dict
Tests
PyTest
Risk and impact
What is the risk level of the change and why?

Minimal

What is the impact if something does go wrong with this PR?

Hey don't we already check for a 404 error? We then check for the empty return with the if person in data and others like that?
https://github.com/zero-sum-seattle/python-mlb-statsapi/blob/c5fae343f9dbda02caa9f320045f55f8380ce213/mlbstatsapi/mlb_dataadapter.py#L126

@KCNilssen we do check for 400 errors in the MLB Adapter, but then we just pass on as a MlbResult with an empty dict.

Hey don't we already check for a 404 error? We then check for the empty return with the if person in data and others like that?
https://github.com/zero-sum-seattle/python-mlb-statsapi/blob/c5fae343f9dbda02caa9f320045f55f8380ce213/mlbstatsapi/mlb_dataadapter.py#L126

https://github.com/zero-sum-seattle/python-mlb-statsapi/blob/404/mlbstatsapi/mlb_api.py#L84-L85
The key checking is too check for when the REST API returns a 200, but leaves the 'person' key an empty list.

This will be nice because it'll be very apparent in our code what the function returns, a object, or list, None, and empty list.


adding ap-southeast-3 region

Description

Adding support for the new region so that the pricing module works again.
Motivation and Context

Adding support for the new region: https://aws.amazon.com/blogs/aws/now-open-aws-asia-pacific-jakarta-region/
Closes #5
Breaking Changes

How Has This Been Tested?

[ x] I have tested and validated these changes using one or more of the provided examples/* projects

I tested with the pricing-resources example

@antonbabenko can you please take a look

This PR is included in version 1.3.0 :tada:


Static content not reloaded on local change

When running static-war example as a Java application from Eclipse, executing the org.wildfly.swarm.container.Container main class, the server starts properly, Howdy is dispayed on http://localhost:8080/, but if the index.html page is updated, the content is not reloaded in the browser.
This is something @bobmcwhirter and I discussed last september. Although the original issue was deleted from github, I still have that quote from Bob:
You can edit your static content and they will appear.
It uses a mixture of file and classloader-based undertow resource handlers,
so that filesystem wins, allowing continual editing within a running app.

So I wonder whether this is something I'm doing wrong with the example or a deeper problem

Opened https://issues.jboss.org/browse/SWARM-230. Will follow up over there.


Can we include video links in README

All the video links mentioned in the youtube video description can we add it on README, it would be very convenient.

@nayabatir1 we can include that. I am also creating a syllabus and CONTRIBUTING.md file. Let me know if you have any further suggestions.

@nayabatir1 we can include that. I am also creating a syllabus and CONTRIBUTING.md file. Let me know if you have any further suggestions.

Sure

@nayabatir1 We have implemented the suggestions. I am closing this issue now. Feel free to raise new Issue again if you have some suggestions for the repo.


[Bug]: Table with and without connected data have different heights

Is there an existing issue for this?

[X] I have searched the existing issues

Description
Table with and without connected data have different heights
Steps To Reproduce

Public Sample App
No response
Environment
Deploy Preview
Severity
High (Blocker to building or releasing)
Issue video log
No response
Version
cloud

Issue still exists cc: @jsartisan


Accepting a plain string as a response

Hello.
I'm very happy of using siesta in my swift built application.
But I need a little help for dealing with a legacy service. This service returns me a plain "true" string in case of positive outcome. But because of this, I'm getting the sequent error:
▿ ResponseInfo
▿ response : Error(userMessage: "Cannot parse server response", httpStatusCode: nil, entity: nil, cause: Optional(Siesta.Error.Cause.JSONResponseIsNotDictionaryOrArray(actualType: "__NSCFBoolean")), timestamp: 507553809.87161499)
▿ Failure : Error
- userMessage : "Cannot parse server response"
- httpStatusCode : nil
- entity : nil
▿ cause : Optional
▿ Some : JSONResponseIsNotDictionaryOrArray
- actualType : "__NSCFBoolean"
- timestamp : 507553809.87161499

isNew : true

Is there a way to configure my  Service, in order to prevent errors in cases like this?
Thank you very much for your work!

You can remove default parsing and apply Siesta’s string transformer if you want to parse all responses for a resource as strings. (See the Pipeline section of the user guide.)
In your case, however, where a JSON service is returning a bare JSON literal instead of a dictionary or array, there's a non-obvious but simple solution described here:
https://github.com/bustoutsolutions/siesta/issues/150
In your case, you want to tell Siesta to consider bare booleans to be valid JSON values:
extension Bool: Siesta.JSONConvertible { }

(The Siesta. is just for clarity.)
This should fix your problem. Feel free to reopen this issue if it doesn’t.

I'll try asap and let you know. For the time being, thank you so much!


String index bug

When I invoke the .PredictLabel method with some inputs, a call to Substring() throws an unhandled exception up the stack.
at System.String.Substring(Int32 startIndex, Int32 length)
   at NW.NGramTextClassification.TextClassifierComponents.<>c.<.cctor>b__31_1(String text, UInt32 length)
   at NW.NGramTextClassification.TextClassifier.PredictLabel(String text, ITokenizationStrategy strategy, INGramTokenizerRuleSet ruleSet, List1 labeledExamples)    at NW.NGramTextClassification.TextClassifier.PredictLabel(String text, List1 labeledExamples)
   at ****.InferReporterType(String input) in ****:line 100

I've dug a little more and this appears to be a bug in the truncation code in LabeledExample.cs:43.  I'm not sure how my input is causing this, but this is the only call to Substring I can find in all the production code.

@projectstann
Thank you for reporting the bug!
Do you have an example of the data you are feeding to the library so that I can recreate the bug?

No problem. To simplify things, I just used the examples from your GitHub page and I get the same error.
public static List GenerateExamples()
{
List<(string label, string text)> tuples = new List<(string label, string text)>()
{
("en", "VerksamhetsbeskrivningGoGift is a company which focuses on [...]"),
("en", "You will report to the Team Manager. You will get the opportunity [...]")
};
        return new LabeledExampleFactory().Create(tuples);
    }

    ITextClassifier textClassifier = new TextClassifier();// new TextClassifier(new TextClassifierComponents(), new TextClassifierSettings(0));
    public string InferReporterType(string input)
    {
        TextClassifierResult result;
        try
        {
            Console.WriteLine("Input: " + input);
            result = textClassifier.PredictLabel(input, labeledExamples);
            Console.WriteLine("Output: " + result.Label);
            return result.Label;
        }
        catch (Exception e)
        {
            Console.Error.WriteLine(e.Message);
        }
        return "Unknown";
    }

The Substring exception is thrown on the call to .PredictLabel.

I'll try to allocate some time to look into it in the next evenings and get back to you! Thanks!

Let me know if I can help you pinpoint the issue. It may be something non-obvious because it looks like you have a lot of unit tests that cover your own basic test cases.

@projectstann
I spent a bunch of hours debugging the issue you found in NW.NGramTextClassification v1.0.0, and I did solve it.
The problem was in DefaultTextTruncatingFunction, which has been rewritten from scratch in order to solve it.
Fixing the bug exposed a pair of additional bugs, which have been fixed as well.
With a test coverage of ca. 92% these bugs were really hard to spot.
All the fixes will be included in the new version of the library that I'm going to push in the next few hours.
Please download it and try it out 😊
The following Program.cs:

using System;
using System.Collections.Generic;
using NW.NGramTextClassification;

namespace NW.NGramTextClassificationClient
{
    class Program
    {
        static void Main(string[] args)
        {

            string label = InferReporterType("some");
            Console.WriteLine(label);

            Console.ReadKey();

        }
        static List<LabeledExample> GenerateExamples()
        {
            List<(string label, string text)> tuples = new List<(string label, string text)>()
            {
                ("en", "VerksamhetsbeskrivningGoGift is a company which focuses on [...]"),
                ("en", "You will report to the Team Manager. You will get the opportunity [...]")
            };

            return new LabeledExampleFactory().Create(tuples);

        }
        static string InferReporterType(string input)
        {

            List<LabeledExample> labeledExamples = GenerateExamples();
            ITextClassifier textClassifier = new TextClassifier();

            TextClassifierResult result;
            try
            {
                Console.WriteLine("Input: " + input);
                result = textClassifier.PredictLabel(input, labeledExamples);
                Console.WriteLine("Output: " + result.Label);
                return result.Label;
            }
            catch (Exception e)
            {
                Console.Error.WriteLine(e.Message);
            }
            return "Unknown";
        }

    }
}

will return the following log:
Input: some
Attempting to predict the label of the provided text...
The following text has been provided: 'some'.
The following 'ITokenizationStrategy' object will be used: '[ Pattern: '[\w0-9_]{1,}', Delimiter: ' ', ToLowercase: 'True' ]'.
The following 'INGramTokenizerRuleSet' object will be used: '[ DoForMonograms: 'True', DoForBigrams: 'True', DoForTrigrams: 'True', DoForFourgrams: 'False', DoForFivegrams: 'False' ]'.
'2' LabeledExample objects have been provided.
The 'DoForBigrams' rule can't be applied to the provided string: 'some'.
Unknown

If you change the input string to some string:
Input: some string
Attempting to predict the label of the provided text...
The following text has been provided: 'some string'.
The following 'ITokenizationStrategy' object will be used: '[ Pattern: '[\w0-9_]{1,}', Delimiter: ' ', ToLowercase: 'True' ]'.
The following 'INGramTokenizerRuleSet' object will be used: '[ DoForMonograms: 'True', DoForBigrams: 'True', DoForTrigrams: 'True', DoForFourgrams: 'False', DoForFivegrams: 'False' ]'.
'2' LabeledExample objects have been provided.
The 'DoForTrigrams' rule can't be applied to the provided string: 'some string'.
Unknown

If you change the input string to some string string:
Input: some string string
Attempting to predict the label of the provided text...
The following text has been provided: 'some string string'.
The following 'ITokenizationStrategy' object will be used: '[ Pattern: '[\w0-9_]{1,}', Delimiter: ' ', ToLowercase: 'True' ]'.
The following 'INGramTokenizerRuleSet' object will be used: '[ DoForMonograms: 'True', DoForBigrams: 'True', DoForTrigrams: 'True', DoForFourgrams: 'False', DoForFivegrams: 'False' ]'.
'2' LabeledExample objects have been provided.
The provided text has been tokenized into '9' INGram object.
Comparing the provided text against the following LabeledExample: '[ Id: '1', Label: 'en', Text: 'Verksamhetsbeskrivni...', TextAsNGrams: '21' ]'...
The calculated 'SimilarityIndex' value is '0'.
The rounded 'SimilarityIndex' value is '0'.
The following SimilarityIndex object has been added to the list: '[ Id: '1', Label: 'en', Value: '0' ]'.
Comparing the provided text against the following LabeledExample: '[ Id: '2', Label: 'en', Text: 'You will report to t...', TextAsNGrams: '36' ]'...
The calculated 'SimilarityIndex' value is '0'.
The rounded 'SimilarityIndex' value is '0'.
The following SimilarityIndex object has been added to the list: '[ Id: '2', Label: 'en', Value: '0' ]'.
The tokenized text has been successfully compared against the provided list of LabeledExample objects.
'2' SimilarityIndex objects have been computed.
The following unique labels have been found in the provided SimilarityIndex list: '[en]'.
Calculating 'SimilarityIndexAverage' for the following label: 'en'...
The calculated 'SimilarityIndexAverage' value is '0'.
The rounded 'SimilarityIndexAverage' value is '0'.
The following SimilarityIndexAverage object has been added to the list: '[ Label: 'en', Value: '0' ]'.
'1' SimilarityIndexAverage objects have been computed.
The following verification has failed: 'ContainsAtLeastOneIndexAverageThatIsNotZero'.
The predicted label is: ''.
The prediction has failed. Try increasing the amount of provided LabeledExample objects.
Output:

UPDATE: NW.NGramTextClassification v1.1.0 is up 😊

Excellent! Thanks for putting in the time to fix it. I'll download the new version soon. 😁.
I'll let you know how it goes.


SVG icons not showing when including the static resources in a page.

Hi, we have built a salesforce (SF) application deploying to SF as a static resource.
We execute it from a Lightning Component like that.
  <lightning:container aura:id="schedulingContainer" class="schedulingContainerWrapper" src="{!$Resource.schedulingComponentVisjs + '/index.html?lightningContainer=true'}"
    onmessage="{!c.handleMessage}" onerror="{!c.handleError}" />

And the entry point of the application is resolved ( because of the ligthing container ) into:
https://pve-alpha-dev-ed--c.container.lightning.com/lcc/1519322045000/schedulingComponentVisjs/app.bundle.js
Also we expose this application from a VF page like that:
<apex:includeScript value="{!URLFOR($Resource.schedulingComponentVisjs, 'app.bundle.js')}"/>

And the entry point from this Page is resolved as:
https://pve-alpha-dev-ed--c.eu11.visual.force.com/resource/1519322045000/schedulingComponentVisjs/app.bundle.js
Everything works in both places ( Lightning Component and VF page  ), even icons from other libraries except with the svg icons used by this design-system-react.
As you can see, Lightning Component works, the SVGs have content inside the tags:

But for the page, executing exactly the static resource and with the same reference in the SVG, doesn't:
.
Just to show the third place, when we work locally, we have those icons also working with the application not yet minified and packaged:

We have configured the IconSettings in the root of our React application (index.js) like:
render(
    <Provider store={store}>
      <IconSettings
        actionSprite={actionSprite}
        customSprite={customSprite}
        doctypeSprite={doctypeSprite}
        standardSprite={standardSprite}
        utilitySprite={utilitySprite}
      >
        <App />
      </IconSettings>
    </Provider>,
    document.getElementById('app')
  );

How could we make those icons to be shown in the **VF page **?

I suspect that the SVG is attempting to load from the wrong URL. The URL for a Visualforce page is "/apex/pageName", so the SVG is probably trying to load from "/apex/svgName.svg" instead of "/resource//svgName.svg".
You can verify my suspicion by looking at the Network tab in the Chrome debugger, and look at the Request URL while loading the SVG file.
If this is right, the solution is probably changing the loader for svg files in your webpack.js. Not quite sure.

I finally got the solution and the explanation. Checking the network panel trying to find a problem when loading the SVG from the page, I found exactly what was happening. Adding the configuration on the as I put previously:
<IconSettings
  actionSprite={actionSprite}
  customSprite={customSprite}
  doctypeSprite={doctypeSprite}
  standardSprite={standardSprite}
  utilitySprite={utilitySprite}
>

The VF Page try to resolve the SVG url to this one:
https://pve-alpha-dev-ed--c.eu11.visual.force.com/apex/eba50582ac0c773b2c45cf502fc7f01c.svg
Instead of the correct one:
https://pve-alpha-dev-ed--c.eu11.visual.force.com/resource/1519322045000/schedulingComponentVisjs/eba50582ac0c773b2c45cf502fc7f01c.svg
Also I noticed that from the LCC it works like a charm resolving the SVG path to this one:
https://pve-alpha-dev-ed--c.eu11.visual.force.com/resource/1519322045000/schedulingComponentVisjs/eba50582ac0c773b2c45cf502fc7f01c.svg
So I tried to pass the static resource URL to the iconPath but again it was not working, because of the sub-folders of each type of icons.
At the end I found looking at the network the solution as it follows.
We have the correct structure of the icons always included in SF orgs in the following path: /_slds/icons.
So the final solution to have the icons working in Salesforce environment (from the .page and from the .cmp ) is just add this simple configuration. And not including the SVG in our static resources but instead consume the icons from Salesforce.
      <IconSettings iconPath="/_slds/icons">
        <App />
      </IconSettings>

I'm sure this information will be very useful to anyone trying to use this library in SF environment.

Thanks @robertovg for following up with the solution. I'm sure you are not the only one. I know I've been asked questions like this in the past.
Although this is a Salesforce project, I'd prefer to keep platform specific question to a minimum and keep issues focused on the code within this project.
For folks that find this thread in the future:

Salesforce platform questions: Please visit https://success.salesforce.com,  https://partners.salesforce.com, or https://salesforce.stackexchange.com I think you will find the experts on these topics more active within those resources.
This project uses SLDS markup, but you should submit HTML/CSS issues to https://github.com/salesforce-ux/design-system.

Thanks, I already tried on those changes but was salesforce miss information than your fault.
Thanks for the support anyway!


Use tag to specify default value of config

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #1392
Does this PR introduce a user-facing change?:

NONE

Nice
/lgtm

Just a quick question, the remaining fields (like PipedGit) gonna be gradually updated to use the tags, right?

Yes. They will be gradually migrated in the future.

got it. there you go!
/approve


Full Scale Performance: Single Process, Sharrow Off, Explicit Chunking

This is the issue to report on memory usage and runtime performance...
data_dir: "data-full" full scale skims (24333 MAZs)
households_sample_size: 0 (full scale 100% sample of households)
sharrow: "false"
multiprocess: false single process
chunk_training_mode: explicit

Run with full sample, sharrow on, and single process (1 TB memory, Intel Xeon Gold 6342 @ 2.8GHz machine).
Runtime is 1428.2 mins (compared to 1,433.1 mins for a similar run with NO chunking)
Chunking is set to explicit in settings_mp.yaml, and explicit chunking is enabled for a number of models whose memory usage was higher than 300Gb in a previous sharrow run with same specs but with NO chunking. These models are:
compute_accessbility
mandatory_tour_scheduling
non-mandatory_tour_scheduling
trip_destination
For these models, the argument explicit_chunk was set to 0.5 in each respective model's yaml file, indicating the need for 2 (=1/0.5) chunks. Further chunks may be added in other tests and as required.
timing_log.csv
activitysim.log
memory_profile.csv

Below is the run without chunking:


Need an option to select encoding when opening file.

Some files have different encoding, and as they are produced as is, so it is preferred to keep them rather than translate them first.

I'm not sure if we really need to select the encoding, since JSON usually is UTF-8.
What we definitely need is proper UTF-8 support with correct handling of multi-byte UTF-8 characters.
Here is a small UTF-8 JSON file that contains some German umlauts and other international charracters: umlauts.json
This is what it's supposed to loke like:
{
	"ascii_only": "abcdefgABCDEFG",
	"german_umlauts": "xäxöxü xÄxÖxÜx ß",
	"other_international": "áéíóú xxx âêîôû xxx àèìòù",
	"umlauts_in_key_äöü_x": "but not in value"
}

BigJsonTreeViewer interprets this as ANSI encoding, which leads to a result that looks like this:
{
	"ascii_only": "abcdefgABCDEFG",
	"german_umlauts": "xÃ¤xÃ¶xÃ¼ xÃ„xÃ–xÃœx ÃŸ",
	"other_international": "Ã¡Ã©ÃÃ³Ãº xxx Ã¢ÃªÃ®Ã´Ã» xxx Ã Ã¨Ã¬Ã²Ã¹",
	"umlauts_in_key_Ã¤Ã¶Ã¼_x": "but not in value"
}

@silmaril42, thanks for the sample!
You're right that UTF-8 should be supported as the JSON RFC requires it.
I don't know off-hand how difficult this would be to do with the MFC controls used by this viewer - hopefully / quite possibly a simple solution exists.
I also can't indicate a time frame that I would be able to look into it. But, since this is the only posted issue so far, it is automatically at the top of the to do list :)
If somebody can help, pull requests are very welcome and it would be easier for me to find time to review a PR than to look into it from scratch. Specific suggestions about how to implement a fix are also welcome.


Using the app on two different devices

What is missing
Use the app on different devices and have warnings appear on both/send test results from both.
Why should it be included
Some people using two or more different devices at different times (such as private and business phones). They might not have each device on them at all times and might miss a warning, or not be able to send a positive test result.
Suggested change

Manually transferring tracing data from one device to the other whenever they are switched (QR code)?
Linking devices to automatically transfer tracing data?
Linking all devices to a central email/phone number to receive warnings, and send test results?
?

Closely related to #82

Dear @KayTeabag,
this is a duplicate of #82 and #171 - as a consequence we will close this issue. The use of multiple devices is currently out of scope. Please see the discussion in #82 for details and also notable aspects concerning this topic.
Thank you for your understanding.
Mit freundlichen Grüßen/Best regards,
SW
Corona Warn-App Open Source Team


Empty id attribute in formgroup

Hi,
if I use a button as example:
@b3.submit('class -> "btn btn-primary btn-block", 'id -> "search"){@page("search")}

then this produces a form group with an empty ID:
<div class="form-group  " id="">
	<button type="submit" class="btn btn-primary btn-block" id="search">Suchen</button>
</div>

This problem occurs also with compounded fields (date range). Normally empty id attributes shouldn't be added, because this leads to validation errors.
Best regards,
Christian

Thank you @akkie. For the issue and specially for Silhouette! ;)
I've just fixed it in new 0.4.1-SNAPSHOT, or I hope so… Now the id is taken from '_id or 'id+_field or it's hidden. So for your example it should take the value "search_field"

You are welcome! Works like a charm! Thanks


Rename std2 to safestd or similar?

Overall, this draft proposal is very interesting and I'm very excited about it. However, I'm not stoked about the idea of the safe standard library being called std2. Could we have a more useful name like safestd or similar?

I like std::safe :)


WKT de-serialization fails when M and Z values are present

If the data has M and Z values, the WKT to Point conversion fails. Changes were required in the WKT reader to fix this. I've already made a pull request for this issue.  https://github.com/cschwarz/wkx-sharp/pull/2

Thank you for your PR!


Update to bitflags 2

Hi! Thank you for your contribution! I apologise for the delay, life happens :-) Looks okay, let's see if it passes the pipelines.

I'll take care of the freebsd CI, thanks again!


code-insiders --help does not mention --log

Refs: #39572
Comand line, code-insiders --help
--log option is not mentioned

Do we log by default?
If yes, then the help should mention the default being used.
If no, then the message should say something like `Enables logging abc to xyz. Allowed levels are ...

Default is info. Will mention that in the description.


Prevent common passwords

There is a service[1] that provides api access to more common cracked passwords.
It is not considered to be good practice to ban these passwords[2].
I therefore suggest we check any new password set against the api.
[1] https://haveibeenpwned.com/
[2] https://www.microsoft.com/en-us/research/wp-content/uploads/2016/06/Microsoft_Password_Guidance-1.pdf

Would this involve sending an unhashed user-entered password through to a foreign API?

There are a couple of methods of access.
I don't suggest using one that requires us to send potential passwords to
them, unless there's a good way to do that.
On 18 December 2017 at 08:15, Salla Karppinen<EMAIL_ADDRESS>wrote:

Would this involve sending an unhashed user-entered password through to a
foreign API?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/oaeproject/Hilary/issues/1491#issuecomment-352354711,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKc2L_nyFDF251dbImkLbK4D9RDTOwBuks5tBh8hgaJpZM4Q52GI
.

There is no good way, which is why I asked. Which method were you thinking of?


xcode 8.1 Swift Compiler Error in ARTrackingManager.swift

if self.userLocation != nil && self.reloadLocationPrevious != nil && self.reloadLocationPrevious!.distance(from: self.userLocation!) > self.reloadDistanceFilter
Binary operator > cannot be applied to operands of type CLLocationDistance (aka Double) and CLLocationDistance!

I can confirm this issue. It appeared after updating XCode to 8.1 and using 10.1 SDK.
Will fix it temporary in pod code.
Hoping for a fix.
Thanks!

Sorry! Updated pods and issue is fixed.