Some validation errors are not being identified

Reported here: datahq/datahub-qa#68
We want to improve showcase page, so it should have error messages as per readme of the dp, eg, when pushing following datapackages some errors are not catched in the frontend:
https://github.com/frictionlessdata/test-data/tree/master/packages/types-formats-and-constraints

string - The value “https:/domain.com” in column “uri” is not type “string” and format “uri”
datetime - The value “2018/01/02T00:00:00” in column “any” is not type “datetime” and format “any”
time - The value “00:01” in column “any” is not type “time” and format “any”

Acceptance criteria
the dataset page should have error messages on showcase page as per readme of the dp:

[x] string - The value “https:/domain.com” in column “uri” is not type “string” and format “uri”

WONTFIX: https://github.com/datahq/frontend/issues/191#issuecomment-370723362

[x] datetime - The value “2018/01/02T00:00:00” in column “any” is not type “datetime” and format “any”

WONTFIX: https://github.com/datahq/frontend/issues/191#issuecomment-370764258

[x] time - The value “00:01” in column “any” is not type “time” and format “any”

WONTFIX:  '00:01' is not time? :face_with_head_bandage:  Really?!! Who's that guy, who created the corresponding test?

Tasks

[ ] add tests for tableschema-py
[ ] fix tableschema-py

Analysis

data validate uses tableschema-js and found all errors
server users tableschema-py and NOT found some errors

How to reproduce

data validate and remember errors
data push and see errors on the dataset page

string:
$ data validate test-data/packages/types-formats-and-constraints/string/
$ data push test-data/packages/types-formats-and-constraints/string/

datetime:
$ data validate test-data/packages/types-formats-and-constraints/datetime/
$ data push test-data/packages/types-formats-and-constraints/datetime/

time:
$ data validate test-data/packages/types-formats-and-constraints/time/
$ data push test-data/packages/types-formats-and-constraints/time/

WONTFIX: String validator accept http:/datahub.io uri
The difference between JS and Python versions of tableschema about validating URI is:

tableschema-js uses "validator": "^9.2.0" from npm
tableschema-py uses https://pypi.python.org/pypi/rfc3986

AND

http:/datahub.io is NOT VALID for npm validator
http:/datahub.io is VALID for pypi rfc3986

When I goes deeper I realized that there is an ANCIENT & GREAT HOLY WAR called:
Accept or not Accept an URI with less or more than two // separators:

https://daniel.haxx.se/blog/2016/05/11/my-url-isnt-your-url/
The WHATWG spec says it has to be one slash and that a parser must accept an indefinite amount of slashes. “http:/example.com” and “http:////////////////////////////////////example.com” are both equally fine. RFC 3986 and many others would disagree.

Also, most of browsers accept  http:/datahub.io uri - I just have checked.
So the pypi implementation of rfc3986 DOES accepts this kind of invalid URI;
And I doubt that @roll will accept any PR where we will implement URI validator by ourselves, just for fixing this particular edge case.
Also, the URI validating REGEX is so f**king complex:
/
# protocol user host-ip port path path path querystring fragment
^
#protocol
(?:(?<scheme>[a-zA-Z][a-zA-Z\d+-.]*):)?
(?:
  (?:
    (?:
        \/\/
        (?:
            #userinfo
            (?:((?:[a-zA-Z\d\-._~\!$&'()*+,;=%]*)(?::(?:[a-zA-Z\d\-._~\!$&'()*+,;=:%]*))?)@)?
            #host-ip
            ((?:[a-zA-Z\d-.%]+)|(?:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:\[(?:[a-fA-F\d.:]+)\]))?
            #port
            (?::(\d*))?
        )
    )
    #slash-path
    (
        (?:\/[a-zA-Z\d\-._~\!$&'()*+,;=:@%]*)*
    )
  )
 #slash-path
 |(\/(?:(?:[a-zA-Z\d\-._~\!$&'()*+,;=:@%]+(?:\/[a-zA-Z\d\-._~\!$&'()*+,;=:@%]*)*))?)
 #path
 |([a-zA-Z\d\-._~\!$&'()*+,;=:@%]+(?:\/[a-zA-Z\d\-._~\!$&'()*+,;=:@%]*)*)
)?
#querystring
(?:\?([a-zA-Z\d\-._~\!$&'()*+,;=:@%\/?]*))?
#fragment
(?:\#([a-zA-Z\d\-._~\!$&'()*+,;=:@%\/?]*))?
$
/x

So I'm leaving this URI validator as it is.

WONTFIX: tableschema-py validator accepts datetime 2018/01/02T00:00:00 format: Any
This 2018/01/02T00:00:00 is a

Valid date due to python lib

>>> from dateutil.parser import parse
>>> parse("2018/01/02T00:00:00")
datetime.datetime(2018, 1, 2, 0, 0)

Invalid date due to JS libs

> moment = require('moment')
> moment("2018/01/02T00:00:00")
Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions.

FIXED:
JS version of tableschema gives some validation errors, while PY version doesn't:

00:01 is not Time
"2018/01/02T00:00:00" is not datetime with format 'Any'
http:/datahub.io is not correct URI
(I agree with the last one, but developers of rfc3986 validator lib for Python are not agree)

So, everything is good with our Pipeline.

If we think that 00:01 is valid time format - we should fix tableschema-js (and data validate) instead.
If we think that 00:01 is invalid time format - then we should validate a dataset prior pushing, coz tableschema-py Authors will not accept such a change anyway.

And I'm going to change the corresponding QA test.


Add an egress rule to allow outgoing traffic to mysql 3306

Need to allow bastion to connect to mysql port 3306. The launch configuration resets the egress rules when autoscaling creates a new instance


Sidebar tweaks

This partially implements a fix for the sidebar overloading CPU in BlazarUI.

We poll every 10 seconds instead of every 5
We don't poll if the tab is not focused

cc @andyhuang91 @gchomatas

👍


[Ambient] Restrict outgoing traffic through an egress gateway

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
I configured the egress gateway to restrict the outgoing traffic following this document. Here are my manifests as follows:

Istio Gateway

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: egressgateway
  namespace: common-gateway
spec:
  selector:
    istio: egressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - "*"
    port:
      name: tls
      number: 443
      protocol: TLS
    tls:
      mode: PASSTHROUGH

K8S Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    istio.io/waypoint-for: all
  name: waypoint
  namespace: istio-egress
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: mesh
    port: 15008
    protocol: HBONE

Istio ServiceEntry

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: genk
  namespace: istio-egress
spec:
  hosts:
  - genk.vn
  ports:
  - name: tls
    number: 443
    protocol: TLS
  resolution: DNS

Istio DestinationRule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-genk
  namespace: istio-egress
spec:
  host: istio-egressgateway.common-gateway.svc.cluster.local
  subsets:
  - name: genk

Istio VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-genk-through-egress-gateway
  namespace: istio-egress
spec:
  hosts:
  - genk.vn
  gateways:
  - mesh
  - common-gateway/istio-egressgateway
  tls:
  - match:
    - gateways:
      - mesh
      port: 443
      sniHosts:
      - genk.vn
    route:
    - destination:
        host: istio-egressgateway.common-gateway.svc.cluster.local
        subset: genk
        port:
          number: 443
  - match:
    - gateways:
      - common-gateway/istio-egressgateway
      port: 443
      sniHosts:
      - genk.vn
    route:
    - destination:
        host: genk.vn
        port:
          number: 443
      weight: 100

K8S NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: istio-egress
  namespace: istio-egress
spec:
  podSelector: {}
  ingress:
  - {}
  egress:
  - ports:
    - protocol: TCP
      port: 15008
  - ports:
    - protocol: TCP
      port: 15012
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: common-gateway
  policyTypes:
  - Ingress
  - Egress

When I curl the external service https://genk.vn, the result failed. Here are the logs as follows:

Pod tools.istio-egress logs

root@tools:/# curl -Iv https://genk.vn
*   Trying <IP_ADDRESS>:443...
* Connected to genk.vn (<IP_ADDRESS>) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to genk.vn:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to genk.vn:443

Pod ztunnel logs

2024-08-06T14:43:38.102580Z     info    access  connection complete     src.addr=<IP_ADDRESS>:34564 src.workload="tools" src.namespace="istio-egress" src.identity="spiffe://cluster.local/ns/istio-egress/sa/default" dst.addr=<IP_ADDRESS>:15008 dst.hbone_addr=<IP_ADDRESS>:443 dst.service="genk.vn" dst.workload="waypoint-574c4b648-zb5bd" dst.namespace="istio-egress" dst.identity="spiffe://cluster.local/ns/istio-egress/sa/waypoint" direction="outbound" bytes_sent=517 bytes_recv=0 duration="10009ms"

Pod waypoint.istio-egress logs

[2024-08-06T14:43:28.093Z] "- - -" 0 UF,URX - - "-" 0 0 10008 - "-" "-" "-" "-" "<IP_ADDRESS>:443" inbound-vip|443|tcp|genk.vn - <IP_ADDRESS>:443 <IP_ADDRESS>:39690 - -

I don't know why the traffic goes to the waypoint in the same namespace istio-egress of pod tools instead of istio-egressgateway.common-gateway as I expected.
Version
$ istioctl -i kube-system version
client version: 1.23-alpha.749dbacd692ea417455ce20d29ba80b7a0cce21b
control plane version: 1.23-alpha.749dbacd692ea417455ce20d29ba80b7a0cce21b
data plane version: 1.23-alpha.749dbacd692ea417455ce20d29ba80b7a0cce21b (12 proxies)

Additional Information
No response

I think the problem is you are mixing up the old sidecar egress approach with the new waypoint approach.
Here is what you need:
Install with --set values.cni.ambient.dnsCapture=true --set values.pilot.env.PILOT_ENABLE_IP_AUTOALLOCATE=true. You need to be using 1.23.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: egress
  namespace: istio-egress
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
---
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: httpbin.org
  namespace: istio-egress
  labels:
    istio.io/use-waypoint: egress
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS

Thats it

@howardjohn It is still not working.
~ $ curl -Iv https://genk.vn
* Host genk.vn:443 was resolved.
* IPv6: 2001:2::2
* IPv4: <IP_ADDRESS>
*   Trying <IP_ADDRESS>:443...
* Connected to genk.vn (<IP_ADDRESS>) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /cacert.pem
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS alert, record overflow (534):
* OpenSSL/3.3.1: error:0A0000C6:SSL routines::packet length too long
* closing connection #0
curl: (35) OpenSSL/3.3.1: error:0A0000C6:SSL routines::packet length too long

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/controller-version: "5"
  creationTimestamp: "2024-08-09T06:50:33Z"
  generation: 1
  labels:
    istio.io/waypoint-for: all
  name: waypoint
  namespace: istio-egress
  resourceVersion: "1174871"
  uid: ba27e932-93d7-468a-94c5-cd918285b1dc
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: mesh
    port: 15008
    protocol: HBONE
status:
  addresses:
  - type: IPAddress
    value: <IP_ADDRESS>
  conditions:
  - lastTransitionTime: "2024-08-09T06:50:33Z"
    message: Resource accepted
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-08-10T06:40:33Z"
    message: Resource programmed, assigned to service(s) waypoint.istio-egress.svc.cluster.local:15008
    observedGeneration: 1
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 0
    conditions:
    - lastTransitionTime: "2024-08-09T06:50:33Z"
      message: No errors found
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-08-09T06:50:33Z"
      message: No errors found
      observedGeneration: 1
      reason: NoConflicts
      status: "False"
      type: Conflicted
    - lastTransitionTime: "2024-08-09T06:50:33Z"
      message: No errors found
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-08-09T06:50:33Z"
      message: No errors found
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: mesh
    supportedKinds: []

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  creationTimestamp: "2024-08-09T06:50:33Z"
  generation: 4
  labels:
    istio.io/use-waypoint: waypoint
  name: istio-egress
  namespace: istio-egress
  resourceVersion: "1193425"
  uid: b0ac4747-fbc0-4529-b05b-215f9f88bd2f
spec:
  hosts:
  - genk.vn
  location: MESH_EXTERNAL
  ports:
  - name: http
    number: 443
    protocol: HTTP
  resolution: DNS
status:
  addresses:
  - host: genk.vn
    value: <IP_ADDRESS>
  - host: genk.vn
    value: 2001:2::2

Logs from waypoint.istio-egress
[2024-08-10T07:08:46.653Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - <IP_ADDRESS>:443 <IP_ADDRESS>:48926 - -

You are sending https but configuring it to be http. You'll either need to configure it to be https, or send http (and then you can add TLS with a destination rule)

@howardjohn, I configured it to be https in the first place which I posted about this issue. That's a problem for me so that why I need you to help me with it.

Your original configuration is trying to configure the legacy egress gateway way so its wrong beyond just the HTTPS part.
You just need
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  labels:
    istio.io/use-waypoint: waypoint
  name: istio-egress
  namespace: istio-egress
spec:
  hosts:
  - genk.vn
  location: MESH_EXTERNAL
  ports:
  - name: https
    number: 443
    protocol: HTTPS
  resolution: DNS

@howardjohn Yeah, I did it too. But it still raises this message
* Host httpbin.org:443 was resolved.
* IPv6: 2001:2::2
* IPv4: <IP_ADDRESS>
*   Trying <IP_ADDRESS>:443...
* Connected to httpbin.org (<IP_ADDRESS>) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /cacert.pem
*  CApath: /etc/ssl/certs
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to httpbin.org:443
* closing connection #0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to httpbin.org:443

I have no idea what I am missing or misconfiguring.

Here are my yaml files.
apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/use-waypoint: waypoint
    istio.io/dataplane-mode: ambient
  name: istio-egress
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    istio.io/waypoint-for: all
  name: waypoint
  namespace: istio-egress
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: mesh
    port: 15008
    protocol: HBONE
---
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: istio-egress
  namespace: istio-egress
  labels:
    istio.io/use-waypoint: waypoint
spec:
  hosts:
  - httpbin.org
  ports:
  - name: https
    number: 443
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: istio-egressgateway
  namespace: istio-egress
spec:
  host: waypoint.istio-egress.svc.cluster.local
  subsets:
  - name: istio-egress
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istio-egress
  namespace: istio-egress
spec:
  hosts:
  - httpbin.org
  gateways:
  - mesh
  - common-gateway/herculesdev9-egressgateway
  tls:
  - match:
    - gateways:
      - mesh
      port: 80
      sniHosts:
      - httpbin.org
    route:
    - destination:
        host: istio-egressgateway.common-gateway.svc.cluster.local
        subset: istio-egress
        port:
          number: 80
  - match:
    - gateways:
      - common-gateway/herculesdev9-egressgateway
      port: 80
      sniHosts:
      - httpbin.org
    route:
    - destination:
        host: httpbin.org
        port:
          number: 80
      weight: 100
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: istio-egress
  namespace: istio-egress
spec:
  podSelector: {}
  ingress:
  - {}
  egress:
  - ports:
    - protocol: TCP
      port: 15008
  - ports:
    - protocol: TCP
      port: 15012
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: common-gateway
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: direct-httpbin-to-egress-gateway
  namespace: istio-egress
spec:
  parentRefs:
  - kind: ServiceEntry
    group: networking.istio.io
    name: istio-egress
  rules:
  - backendRefs:
    - name: waypoint
      port: 15008
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: forward-httpbin-from-egress-gateway
  namespace: istio-egress
spec:
  parentRefs:
  - name: waypoint
  hostnames:
  - httpbin.org
  rules:
  - backendRefs:
    - kind: Hostname
      group: networking.istio.io
      name: httpbin.org
      port: 443
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: originate-tls-for-httpbin-org
  namespace: istio-egress
spec:
  host: httpbin.org
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE

@laiminhtrung1997 this is the config you need. Note the 4 configs I have removed
apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/use-waypoint: waypoint
    istio.io/dataplane-mode: ambient
  name: istio-egress
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    istio.io/waypoint-for: all
  name: waypoint
  namespace: istio-egress
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: mesh
    port: 15008
    protocol: HBONE
---
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: istio-egress
  namespace: istio-egress
  labels:
    istio.io/use-waypoint: waypoint
spec:
  hosts:
  - httpbin.org
  ports:
  - name: https
    number: 443
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
# apiVersion: networking.istio.io/v1alpha3
# kind: DestinationRule
# metadata:
#   name: istio-egressgateway
#   namespace: istio-egress
# spec:
#   host: waypoint.istio-egress.svc.cluster.local
#   subsets:
#   - name: istio-egress
# ---
# apiVersion: networking.istio.io/v1alpha3
# kind: VirtualService
# metadata:
#   name: istio-egress
#   namespace: istio-egress
# spec:
#   hosts:
#   - httpbin.org
#   gateways:
#   - mesh
#   - common-gateway/herculesdev9-egressgateway
#   tls:
#   - match:
#     - gateways:
#       - mesh
#       port: 80
#       sniHosts:
#       - httpbin.org
#     route:
#     - destination:
#         host: istio-egressgateway.common-gateway.svc.cluster.local
#         subset: istio-egress
#         port:
#           number: 80
#   - match:
#     - gateways:
#       - common-gateway/herculesdev9-egressgateway
#       port: 80
#       sniHosts:
#       - httpbin.org
#     route:
#     - destination:
#         host: httpbin.org
#         port:
#           number: 80
#       weight: 100
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: istio-egress
  namespace: istio-egress
spec:
  podSelector: {}
  ingress:
  - {}
  egress:
  - ports:
    - protocol: TCP
      port: 15008
  - ports:
    - protocol: TCP
      port: 15012
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: common-gateway
  policyTypes:
  - Ingress
  - Egress
---
# apiVersion: gateway.networking.k8s.io/v1
# kind: HTTPRoute
# metadata:
#   name: direct-httpbin-to-egress-gateway
#   namespace: istio-egress
# spec:
#   parentRefs:
#   - kind: ServiceEntry
#     group: networking.istio.io
#     name: istio-egress
#   rules:
#   - backendRefs:
#     - name: waypoint
#       port: 15008
# ---
# apiVersion: gateway.networking.k8s.io/v1
# kind: HTTPRoute
# metadata:
#   name: forward-httpbin-from-egress-gateway
#   namespace: istio-egress
# spec:
#   parentRefs:
#   - name: waypoint
#   hostnames:
#   - httpbin.org
#   rules:
#   - backendRefs:
#     - kind: Hostname
#       group: networking.istio.io
#       name: httpbin.org
#       port: 443
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: originate-tls-for-httpbin-org
  namespace: istio-egress
spec:
  host: httpbin.org
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE

Note depending on the version, you may be running into two bugs:

https://github.com/istio/istio/issues/52746 - if traffic is failing entirely its probably this one. You can see with the <IP_ADDRESS> here:

$ istioctl pc l waypoint-7f8df594db-grxcl --waypoint
LISTENER                  CHAIN                            MATCH                  DESTINATION
envoy://connect_originate                                  ALL                    Cluster: connect_originate
envoy://main_internal     inbound-vip|443||httpbin.org-tcp ip=<IP_ADDRESS> -> port=443 Cluster: inbound-vip|443|tcp|httpbin.org

https://github.com/istio/istio/pull/52747 - portLevelSettings does not work in waypoint. I would just remove it and make it like this:

spec:
  host: httpbin.org
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    tls:
      mode: SIMPLE

Note you also need --set values.pilot.env.PILOT_ENABLE_IP_AUTOALLOCATE=true --set values.cni.ambient.dnsCapture=true

@howardjohn That's great. I very much appreciate it. Let me check whether I got these bugs.

@howardjohn I configured exactly everything you commented but the traffic still failed.
* IPv6: 2001:2::1
* IPv4: <IP_ADDRESS>
*   Trying <IP_ADDRESS>:443...
* Connected to httpbin.org (<IP_ADDRESS>) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /cacert.pem
*  CApath: /etc/ssl/certs
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to httpbin.org:443
* closing connection #0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to httpbin.org:443

$ kubectl -n kube-system logs -f ds/ztunnel | grep istio-egress
2024-08-20T07:40:25.262643Z     info    access  connection complete     src.addr=<IP_ADDRESS>:52180 src.workload="sleep-5577c64d7c-vk2rd" src.namespace="istio-egress" src.identity="spiffe://cluster.local/ns/istio-egress/sa/sleep" dst.addr=<IP_ADDRESS>:15008 dst.hbone_addr=<IP_ADDRESS>:443 dst.service="httpbin.org" dst.workload="waypoint-5749789b5-rmj65" dst.namespace="istio-egress" dst.identity="spiffe://cluster.local/ns/istio-egress/sa/waypoint" direction="outbound" bytes_sent=517 bytes_recv=0 duration="10001ms"

[2024-08-20T07:40:15.263Z] "- - -" 0 UF,URX - - "-" 0 0 9997 - "-" "-" "-" "-" "<IP_ADDRESS>:443" inbound-vip|443|tcp|httpbin.org - <IP_ADDRESS>:443 <IP_ADDRESS>:44594 - -

That sounds like the waypoint cannot call httpbin at <IP_ADDRESS>:443. That doesn't seem likely an Istio misconfig, but lack of network connectivity. Can you do curl https://httpbin.org fromthe waypoint?

Your
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: istio-egress
  namespace: istio-egress

doesn't allow the waypoint itself to call external

@howardjohn Hix. You're right. That was my silly mistake. The networkPolicy blocks the outbound of the waypoint. When I separated them, the log was much better but it still failed.
$ kubectl -n istio-egress exec -it deploy/sleep -- sh
~ $ curl -Iv https://httpbin.org
* Host httpbin.org:443 was resolved.
* IPv6: 2001:2::2
* IPv4: <IP_ADDRESS>
*   Trying <IP_ADDRESS>:443...
* Connected to httpbin.org (<IP_ADDRESS>) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /cacert.pem
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS alert, record overflow (534):
* OpenSSL/3.3.1: error:0A0000C6:SSL routines::packet length too long
* closing connection #0
curl: (35) OpenSSL/3.3.1: error:0A0000C6:SSL routines::packet length too long

I close it because I can configure the outbound in ambient mesh with @howardjohn's support. I very much appreciate it.

https://github.com/istio/istio/discussions/52727#discussioncomment-10476752 may help @rezaebrahimi1

@howardjohn it only works for me from ns istio-egress. Traffic from other ns (in mesh) goes directly.
Could it be related to calico ebpf-mode?

I close the question, it was necessary to restart the src-pods


Fix bug where SpecularMode DISABLED is not cached

Bug was unreported. But we have 5 SpecularModes, and we were caching them using only 2 bits...
CC @akien-mga
edit:  introduced by https://github.com/godotengine/godot/commit/e577c5b0705168177943fcdf9a0b66c1f8f864f3

Thanks!


Adding Login/Signup using Social Media and also annonymous login in Wiki Commons app

We can use the email of user as the username as will be a better user experience because it is difficult for the user to remember his/her username for different applications.

Does the desktop Commons website allow all of this?

We can add this Oauth feature in the website as well.

Hello, the best solution would be to allow them to signup using their emails and keep them signed in.
And, can I to contribute to the project?

This is similar to #1095 - I believe the answer to both is "unlikely, due to WMF policies".

Yep, unlikely.

Closing this, then. I suppose we can leave #1095 up for now.


Re-work CSS parsing logic.

 Update parsing to start with anchor()

Putting this on hold for now, while we reconsider our approach to parsing stylesheets (see: https://oddbird.slack.com/archives/C03K136KX2A/p1661273459265879).

@jgerigmeyer what's most helpful for me to understand or review here? Is a functional review helpful, or is there something I should be digging into code-wise?

@jgerigmeyer what's most helpful for me to understand or review here? Is a functional review helpful, or is there something I should be digging into code-wise?

@mirisuzanne There's not a lot functional to review yet (other than looking at what's logged to the console, and comparing that with the demo stylesheets). You could skim the code (mostly the changes in parse.ts and fetch.ts), and/or take a look at the console logged data object (which should represent an extraction of the relevant data from the stylesheets).

@sanajaved7 @mirisuzanne I think this is ready for review, then merge into #19?


PANIC: no entry found for key in query-engine/prisma-models/src/record.rs:67:33

Bug description
Invalid `prisma.product.findFirst()` invocation:

  PANIC: no entry found for key in query-engine/prisma-models/src/record.rs:67:33

This is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.

When query data from database at runtime
How to reproduce
It's happen randomly to resolve I've to redeploy layer that contain Prisma engine
Expected behavior
No response
Prisma information
Prisma query happen randomly
Environment & setup

OS: Linux on AWS Lambda
Database: PostgreSQL
Node.js version: 14 on Lambda

Prisma Version
3.13.0

Can give us any additional information? What does the product model in the schema look like? Is there any weird data in the database?

Data Model
model Product {
  id                                           String                                        @id @default(uuid())
  shopId                                       String
  brandId                                      String?
  brand                                        ProductBrand?                                 @relation(fields: [brandId], references: [id])
  type                                         ProductType                                   @default(STOCK)
  internalId                                   String?
  accountingId                                 String?
  sku                                          String?
  titleId                                      String?                                       @unique
  title                                        Translation?                                  @relation(name: "Product.title", fields: [titleId], references: [id])
  descriptionId                                String?                                       @unique
  description                                  Translation?                                  @relation(name: "Product.description", fields: [descriptionId], references: [id])
  slugId                                       String?                                       @unique
  slug                                         Translation?                                  @relation(name: "Product.slug", fields: [slugId], references: [id])
  weight                                       Float?
  weightUnit                                   WeightUnit?
  availableFrom                                DateTime?
  productMedia                                 ProductMedia[]
  options                                      ProductOption[]
  optionValues                                 ProductOptionValue[]
  variants                                     ProductVariant[]
  productCollection                            ProductToCollection[]
  requestQuotationEnabled                      Boolean                                       @default(false)
  bulkOrderEnabled                             Boolean                                       @default(false)
  measurements                                 ProductMeasurement[]
  createdAt                                    DateTime                                      @default(now())
  updatedAt                                    DateTime                                      @updatedAt
  createdBy                                    String
  updatedBy                                    String
  deletedAt                                    DateTime?
  childProducts                                Product[]                                     @relation(name: "ProductToPackage", references: [id])
  parentProducts                               Product[]                                     @relation(name: "ProductToPackage", references: [id])
  salesChannelProducts                         SalesChannelProduct[]
  salesChannelProductPackages                  SalesChannelProductPackage[]
  productAttributes                            ProductAttribute[]
  inventory                                    ProductInventory[]
  productImagePlaceholderPositionAndDimensions ProductImagePlaceholderPositionAndDimension[]
  lastExportedCustomImage                      DateTime?
  meta                                         Json?
  salesChannelProductVariants                  SalesChannelProductVariant[]

  @@unique([id, shopId])
  @@index([brandId])
}

As I checked on Our DB, There is no any weird data.
For the time being, it seems work after we re-install prisma layer to AWS lambda function.

This is the simple query
this.db.product.findFirst({
      where: {
        id,
        shopId,
        deletedAt: null,
      },
    })

@janpio @annibuliful Added data model

@janpio @Weakky I have a question about this issue. Is it possible this error may cause by the pagination cursor?

Everything is possible - only when we looked at it in more detail we can potentially say more. Add as much information as you have available to make it as easy as possible for us to reproduce. Thanks.

@janpio Sure I'll try to collect more information and reproduce step if it happens again.

I've found I think it's a root clause variants relation that has ~7800 records inside of it. Not sure that can clause this issue to happen.

Hey @wiput1999,
I'm unfortunately unable to reproduce your issue. Given where it panics it looks like it's related to some ordering, which might indeed be implicitly added because of some cursor pagination.
Your client snippet doesn't have such pagination though. Are you sure it is this query that triggers the panic?

Hi @Weakky ,
Thank you for following up on this issue. It never happened again since that day not sure why. Other things come to my mind I would like to ask. Is it possible that AWS Lambda running out of memory on runtime can cause this?

Is it possible that AWS Lambda running out of memory on runtime can cause this?
This seems very very unlikely. You'd most likely get an OS error or something else if that was the case, not a user-created rust panic.

The panic happens because the QE tries to order by (in-memory) a field that does not exist. Given that we have no other issue with that panic and that you unfortunately can't reproduce it either, I'm gonna close your issue for now since there's not much more we can do.
If it ever happens again, please re-open that issue and let us know as much as you can.
Thanks for reporting 🙏


TBI - Add LocalStorageConnector to html files for preview to keep personalizations across preview sessions

When previewing apps in WebIDE personalization settings of the developer where persisted across browser refresh and preview sessions using the sessionStorageConnector LocalStorageConnector and storage in browser. This feature was and is not enabled in SAP Fiori apps generated with SAP Fiori tools. When refreshing the browser window all personalizations are resetted, which is inconvenient.
Scope:
add {"connector":"SessionStorageConnector LocalStorageConnector"} to the data-sap-ui-flexibilityServices bootstrap settings in flpSandbox.html files for preview on generation - this is only for preview not for productive delivery
Value
Value
Feature parity with WebIDE.
Architecture Elaboration
Does it requires architecture elaboration?
Notes
Tasks

[ ] Task 1

In addition to the writers, please apply the fix as well at https://github.com/SAP/open-ux-tools/blob/main/packages/preview-middleware/templates/flp/sandbox.html.

Flexibility colleagues adviced that the SessionStorageConnector is the wrong connector. We should use LocalStorageConnector. @zdravko-georgiev

Done


Add AWS ALB to the predefined logs format

Hello,
We are considering to parse AWS ALB - Application Load Balancer logs. It's log format is described in the documentation - Access logs for your Application Load Balancer.
It was already discussed recently

Log format for Amazon LoadBalancer #793
Amazon Application Load Balancer (alb) access log format #797
Trouble parsing AWS ALB logs #1216
how do i process aws elb json formatted logs #1832

But, looks like it is not defined yet in the list of the predefined logs format.
We are mostly using ALB instead of the ELB. It can be usefully to have such an option to simplify parsing like we do it now for CloudFront
--log-format CLOUDFRONT
--log-format AWSALB

An example of the ALB log:
h2 2021-12-19T11:22:20.356787Z app/staging/cb87d24460bf11ec <IP_ADDRESS>:62611 <IP_ADDRESS>:443 0.006 0.008 0.000 200<PHONE_NUMBER> "GET https://service.staging.domain.tld:443/ HTTP/2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/service-staging/946e110660bf11ec "Root=1-30c170ac-60c011ec86070242ac130002" "service.staging.domain.tld" "arn:aws:acm:us-east-1:123456789012:certificate/8194baa8-60bf-11ec-8607-0242ac130002" 290 2021-12-19T11:22:20.342000Z "forward" "-" "-" "<IP_ADDRESS>:443" "200" "-" "-"

And log-format,  date-format and time-format we used from your comment:
zcat *.gz \
| docker run \
    --interactive \
    --rm \
    --env LANG=$LANG \
    allinurl/goaccess \
    --date-spec=hr \
    --real-os \
    --agent-list \
    --output html \
    --log-format='%^ %dT%t.%^ %v %h:%^ %^ %^ %T %^ %s %^ %^ %b "%r" "%u" %^' \
    --date-format=%Y-%m-%d \
    --time-format=%T \
    - > report.html

Looks like it works as expected.
Thank you!

Thanks for sharing that. I've added the format as --log-format=AWSALB . Also added %k %K to cover the TLS fields. e.g.,
goaccess access.log --log-format='%^ %dT%t.%^ %v %h:%^ %^ %^ %T %^ %s %^ %^ %b "%r" "%u" %k %K %^' --date-format=%Y-%m-%d --time-format=%T

Closing this. Feel free to reopen it if needed.


Require autoxtrabackup to be run under root or sudo

I got the error with recent changes:
$ autoxtrabackup
Traceback (most recent call last):
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/bin/autoxtrabackup", line 11, in <module>
    load_entry_point('mysql-autoxtrabackup==1.4.9', 'console_scripts', 'autoxtrabackup')()
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.4.9-py3.5.egg/autoxtrabackup.py", line 170, in all_procedure
  File "/home/shahriyar.rzaev/Python_interpreters/Python_3_5_3/lib/python3.5/logging/handlers.py", line 150, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/home/shahriyar.rzaev/Python_interpreters/Python_3_5_3/lib/python3.5/logging/handlers.py", line 57, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/home/shahriyar.rzaev/Python_interpreters/Python_3_5_3/lib/python3.5/logging/__init__.py", line 1009, in __init__
    StreamHandler.__init__(self, self._open())
  File "/home/shahriyar.rzaev/Python_interpreters/Python_3_5_3/lib/python3.5/logging/__init__.py", line 1038, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/var/log/autoxtrabackup.log'

So it should print the --help if no options passed.
The thing is autoxtrabackup should run with root or sudo, because it is using xtrabackup and doing some sort of directory works.

I have changed my mind on forcing global root/sudo usage, because it will break remote interpreter execution and some other scripts which is crucial for development process.
Rather, added inside code:
$ autoxtrabackup
[Errno 13] Permission denied: '/var/log/autoxtrabackup.log' Please consider to run as root or sudo

$ autoxtrabackup --backup
[Errno 13] Permission denied: '/var/log/autoxtrabackup.log' Please consider to run as root or sudo

Can be merged now.


A new workable deployment yaml for bosh-init

Nice.
@karafelix can we make sure the pipeline is indeed using this (or modified version) and fix any changes you make.
Thanks all. Cheers :beers:

I'm waiting on bosh-init to run, but it seems to have gotten farther than I've ever gotten before, so things are looking positive. I'll submit a pull request with the changes I made (commented above) if they work.


Further development?

Postal looks fantastic, and the approach of interfacing with libetpan directly is the way to go. However I'm saddened to see there's been no activity since 2017 - is this project still supported by its creators? MailCore stopped working recently and I'm saddened to see another great effort has stopped as well.

Hello @guidedways thank you.
I think it's a good think to wrap libetpan from swift even if it's a bit tough to work with. Mailcore was doing the same but in objc and the swift usage was not very idiomatic. I don't know if libetpan is still maintained though.
As for the development, actually it's been a long time that I am no more part of the snips company and my friend and co-developer on this lib left the company too a few years ago. The company does not exist anymore because it was bought by sonos.
I can't act on the project to give it to a new maintainer, can't "archive" it and can't even commit to mark that it is no longer maintained... so this project will be left as-is I guess.
I invite anyone wanting to update/maintain it to create a fork :(
Sorry,
J.

Thanks @jeremiegirault for sharing an update with us, much appreciated. I believe all these (good) libraries have effectively gone dead but I see libetpan and MailCore2 got a patch two days ago for M1. We'll do as you say - fork and update.

If you decide to fork & update it, maybe post the link of your fork in this thread.
I think some other people may be interested in it and I'll take a look at your work :-) .

Agreed @guidedways - please post your fork. I would contribute.

Has anyone gotten this to work with an Apple Silicon machine?


search results have broken links

when searching, the results have broken links, due to trying to connect to a "local host".

Search is using https://lunrjs.com/ and I suspect it has not been properly set up, but is just reading in results from this file: https://github.com/noaa-fisheries-integrated-toolbox/resources/blob/faf4f8e3f4a889fb21aece3a6dc18de63e1be221/assets/js/lunr/lunr-store.js
So it is not providing actual site search results.

This provides some advice on how to get lunr working with jekyll static sites: https://rayhightower.com/blog/2016/01/04/how-to-make-lunrjs-jekyll-work-together/


Legacy behaviour of the LlamaTokenizer and model weight not found

System Info
root@tgi-test-:/data/model# text-generation-launcher -e
2023-12-19T11:48:17.724197Z  INFO text_generation_launcher: Runtime environment:
Target: x86_64-unknown-linux-gnu
Cargo version: 1.70.0
Commit sha: d077150eb7b9884f33e60abe99fe6359bf6e3e64
Docker label: sha-d077150
nvidia-smi:
Tue Dec 19 11:48:17 2023
   +-----------------------------------------------------------------------------+
   | NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.1     |
   |-------------------------------+----------------------+----------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |                               |                      |               MIG M. |
   |===============================+======================+======================|
   |   0  NVIDIA H100 80G...  On   | 00000000:C3:00.0 Off |                    0 |
   | N/A   26C    P0    72W / 700W |      0MiB / 81559MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+

   +-----------------------------------------------------------------------------+
   | Processes:                                                                  |
   |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
   |        ID   ID                                                   Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   +-----------------------------------------------------------------------------+

...
root@tgi-test-:/data/model# text-generation-launcher -V
text-generation-launcher 1.3.3

Information

[ ] Docker
[X] The CLI directly

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
I want to launch the Phind CodeLlama 34B AWQ Model via the commmand:
text-generation-launcher --model-id=$MODEL_PATH --num-shard=2 --port=80 --max-best-of=1  --max-input-length=4096 --max-batch-prefill-tokens=4096 --max-total-tokens=9192 --json-output --validation-workers 4 --quantize=awq

Unfortunatly, the server fails to start with the following log:
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and
thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
You are using a model of type llama to instantiate a model of type . This is not supported for all configurations of models and can yield errors.

and
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 63, in get_filename
    raise RuntimeError(f"weight {tensor_name} does not exist")

RuntimeError: weight model.layers.0.self_attn.q_proj.weight does not exist
 rank=0

Complete Log

root@tgi-test-:/data/model# text-generation-launcher --model-id=/data/model/
2023-12-19T11:47:16.868204Z  INFO text_generation_launcher: Args { model_id: "/data/model/", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "tgi-test-", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma:
None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }
2023-12-19T11:47:16.868350Z  INFO download: text_generation_launcher: Starting download process.
2023-12-19T11:47:21.018593Z  INFO text_generation_launcher: Files are already present on the host. Skipping download.
2023-12-19T11:47:21.572693Z  INFO download: text_generation_launcher: Successfully downloaded weights.
2023-12-19T11:47:21.572925Z  INFO shard-manager: text_generation_launcher: Starting shard rank=0
2023-12-19T11:47:28.416885Z ERROR text_generation_launcher: Error when initializing model
Traceback (most recent call last):
File "/opt/conda/bin/text-generation-server", line 8, in
sys.exit(app())
File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in call
return get_command(self)(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main
return _main(
File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main
rv = self.invoke(ctx)
File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper
return callback(**use_params)  # type: ignore
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 89, in serve
server.serve(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 235, in serve
asyncio.run(
File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
self.run_forever()
File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)

File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 196, in serve_inner
model = get_model(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/init.py", line 233, in get_model
return FlashLlama(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_llama.py", line 69, in init
model = FlashLlamaForCausalLM(config, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 412, in init
self.model = FlashLlamaModel(config, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 350, in init
[
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 351, in
FlashLlamaLayer(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 287, in init
self.self_attn = FlashLlamaAttention(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 171, in init
self.query_key_value = load_attention(config, prefix, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 95, in load_attention
return _load_gqa(config, prefix, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 118, in _load_gqa
weight = weights.get_multi_weights_col(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 220, in get_multi_weights_col
w = [self.get_sharded(f"{p}.weight", dim=0) for p in prefixes]
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 220, in
w = [self.get_sharded(f"{p}.weight", dim=0) for p in prefixes]
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 112, in get_sharded
filename, tensor_name = self.get_filename(tensor_name)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 63, in get_filename
raise RuntimeError(f"weight {tensor_name} does not exist")
RuntimeError: weight model.layers.0.self_attn.q_proj.weight does not exist

2023-12-19T11:47:29.179264Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and
thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
You are using a model of type llama to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
Traceback (most recent call last):
File "/opt/conda/bin/text-generation-server", line 8, in
sys.exit(app())
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 89, in serve
server.serve(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 235, in serve
asyncio.run(
File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 196, in serve_inner
model = get_model(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/init.py", line 233, in get_model
return FlashLlama(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_llama.py", line 69, in init
model = FlashLlamaForCausalLM(config, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 412, in init
self.model = FlashLlamaModel(config, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 350, in init
[
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 351, in
FlashLlamaLayer(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 287, in init
self.self_attn = FlashLlamaAttention(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 171, in init
self.query_key_value = load_attention(config, prefix, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 95, in load_attention
return _load_gqa(config, prefix, weights)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 118, in _load_gqa
weight = weights.get_multi_weights_col(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 220, in get_multi_weights_col
w = [self.get_sharded(f"{p}.weight", dim=0) for p in prefixes]
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 220, in
w = [self.get_sharded(f"{p}.weight", dim=0) for p in prefixes]
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 112, in get_sharded
filename, tensor_name = self.get_filename(tensor_name)
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 63, in get_filename
raise RuntimeError(f"weight {tensor_name} does not exist")
RuntimeError: weight model.layers.0.self_attn.q_proj.weight does not exist
rank=0
2023-12-19T11:47:29.278001Z ERROR text_generation_launcher: Shard 0 failed to start
2023-12-19T11:47:29.278017Z  INFO text_generation_launcher: Shutting down shards
Error: ShardCannotStart

Searching for the issue revealed that #1028 #1188 and #1322 have similar problems. The proposed fixes (latest update & DISABLE_EXLLAMA=True) coudn't fix the problem, but there model and quantization is diferent. Hence, i opening the issue
Can you help me?
Thanks in advance
Best
Christoph
Expected behavior
The model should load as expected. My previous setup with Text-generation-inference v1.1.0 has worked perfectly.

In the stacktrace you linked, it seems that quantize is None. Are you sure you correctly set --quantize awq argument or QUANTIZE=awq env var?

Thats true for the above example. Sorry for that. Nethertheless for:
text-generation-launcher --model-id=$MODEL_PATH --num-shard=1 --port=80 --max-best-of=1  --max-input-length=4096 --max-batch-prefill-tokens=4096 --max-total-tokens=9192 --json-output --validation-workers 4 --quantize=awq

The Log is:
root@tgi-test:/usr/src# text-generation-launcher --model-id=$MODEL_PATH --num-shard=1 --port=80 --max-best-of=1  --max-input-length=4096 --max-batch-prefill-tokens=4096 --max-total-tokens=9192 --json-output --validation-workers 4 --quantize=awq
{"timestamp":"2023-12-20T14:42:22.580668Z","level":"INFO","fields":{"message":"Args { model_id: \"/data/model/\", revision: None, validation_workers: 4, sharded: None, num_shard: Some(1), quantize: Some(Awq), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of:  1, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 4096, max_total_tokens: 9192, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"tgi-test-deployment-bf6c6864b-7pn84\", port: 80, shard_uds_path: \"/tmp/text-generatio-n-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: n
None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
{"timestamp":"2023-12-20T14:42:22.580813Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
{"timestamp":"2023-12-20T14:42:26.536273Z","level":"INFO","fields":{"message":"Files are already present on the host. Skipping download.\n"},"target":"text_generation_launcher"}
{"timestamp":"2023-12-20T14:42:27.085384Z","level":"INFO","fields":{"message":"Successfully downloaded weights."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
{"timestamp":"2023-12-20T14:42:27.085833Z","level":"INFO","fields":{"message":"Starting shard"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:42:37.094492Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:42:47.102886Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:42:57.110153Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:43:07.117293Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:43:17.124626Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:43:27.131892Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:43:37.139003Z","level":"INFO","fields":{"message":"Waiting for shard to be ready..."},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:43:46.141708Z","level":"ERROR","fields":{"message":"Error when initializing model\nTraceback (most recent call last):\n  File \"/opt/conda/bin/text-generation-server\", line 8, in <module>\n    sys.exit(app())\n  File \"/opt/conda/lib/python3.10/site-packages/typer/main.py\", line 311, in __call__\n    return get_command(self)(*args, **kwargs)\n  File \"/opt/conda/lib/python3.10/site-packages/click/core.py\", line 1157, in __call__\n    return self.main(*args, **kwargs)\n  File \"/opt/conda/lib/python3.10/site-packages/typer/core.py\", line 778, in main\n    return _main(\n  File \"/opt/conda/lib/python3.10/site-packages/typer/core.py\", line 216, in _main\n    rv = self.invoke(ctx)\n  File \"/opt/conda/lib/python3.10/site-packages/click/core.py\", line 1688, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n  File \"/opt/conda/lib/python3.10/site-packages/click/core.py\", line 1434, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n  File \"/opt/conda/lib/python3.10/site-packages/click/core.py\", line 783, in invoke\n    return __callback(*args, **kwargs)\n  File \"/opt/conda/lib/python3.10/site-packages/typer/main.py\", line 683, in wrapper\n
 return callback(**use_params)  # type: ignore\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py\", line 89, in serve\n    server.serve(\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py\", line 235, in serve\n    asyncio.run(\n  File \"/opt/conda/lib/python3.10/asyncio/runners.py\", line 44, in run\n    return loop.run_until_complete(main)\n  File \"/opt/conda/lib/python3.10/asyncio/base_events.py\", line 636, in run_until_complete\n    self.run_forever()\n  File \"/opt/conda/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever\n
  self._run_once()\n  File \"/opt/conda/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once\n    handle._run()\n  File \"/opt/conda/lib/python3.10/asyncio/events.py\", line 80, in _run\n    self._context.run(self._callback, *self._args)\n> File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py\", line 196, in serve_inner\n    model = get_model(\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py\", line 233, in get_model\n    return FlashLlama(\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_llama.py\", line 65, in __init__\n    weights = Weights(filenames, device, dtype, process_group=self.process_group)\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py\", line 23, in __init__\n    with safe_open(filename, framework=\"pytorch\") as f:\nFileNotFoundError: No such file or directory: \"model-00002-of-00002.safetensors\"\n"},"target":"text_generation_launcher"}
{"timestamp":"2023-12-20T14:43:46.946922Z","level":"ERROR","fields":{"message":"Shard complete standard error output:\n\nYou are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\nYou are using a model of type llama to instantiate a model of type . This is not supported for all configurations of models and can yield errors.\nTraceback (most recent call last):\n\n  File \"/opt/conda/bin/text-generation-server\", line 8, in <module>\n    sys.exit(app())\n\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py\", line 89, in serve\n    server.serve(\n\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py\", line 235, in serve\n    asyncio.run(\n\n  File \"/opt/conda/lib/python3.10/asyncio/runners.py\", line 44, in run\n    return loop.run_until_complete(main)\n\n  File \"/opt/conda/lib/python3.10/asyncio/base_events.py\", line 649, in run_until_complete\n    return future.result()\n\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py\", line 196, in serve_inner\n    model = get_model(\n\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py\", line 233, in get_model\n    return FlashLlama(\n\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_llama.py\", line 65, in __init__\n    weights = Weights(filenames, device, dtype, process_group=self.process_group)\n\n  File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py\", line 23, in __init__\n    with safe_open(filename, framework=\"pytorch\") as f:\n\nFileNotFoundError: No such file or directory: \"model-00002-of-00002.safetensors\"\n"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2023-12-20T14:43:47.038364Z","level":"ERROR","fields":{"message":"Shard 0 failed to start"},"target":"text_generation_launcher"}
{"timestamp":"2023-12-20T14:43:47.038388Z","level":"INFO","fields":{"message":"Shutting down shards"},"target":"text_generation_launcher"}
Error: ShardCannotStart

Maybee i miss something here?


MemoryError: memory allocation failed

when i run code to generate two  bytearrays
w = 800
h = 480

fb_size = int(w * h / 8)
frame_black = bytearray(fb_size)
frame_red = bytearray(fb_size)

got error:
File "<stdin>", line 5, in <module>
MemoryError: memory allocation failed, allocating 48000 bytes

discussion on micropython repo(#10020): Understanding how Micropython interacts with ESP32 ROM, SRAM, SRAM in RTC, and PSRAM
and  Espressif guide - Minimizing RAM Usage

for now, a naive solution is use esp32-s3 or esp32 wrover which has spiram


Google Analytics: Force a new session every 500 events (see #40)

I tested this over the last couple of days. Seems to be working well! Also updated universal-analytics version to 0.4.13.

Sweet. New version published to npm.


Worker deployments exist after worker records are removed

When deploying manageiq in openshift or kubernetes it's possible that worker deployments are left in the namespace when the worker rows have been deleted.
This means that the server no longer knows these workers are running so it also doesn't know to bring them down.
Here's an example:
$ oc get deploy | grep 1-
1-amazon-agent-coordinator-0              1/1       1            1           6h24m
1-amazon-cloud-event-catcher-135          1/1       1            1           73m
1-amazon-cloud-event-catcher-144          1/1       1            1           73m
1-amazon-cloud-metrics-collector          2/2       2            2           41h
1-amazon-cloud-refresh-10-8-9             0/1       1            0           2d19h
1-amazon-cloud-refresh-135-136-137        1/1       1            1           6h25m
1-amazon-cloud-refresh-144-145-146        1/1       1            1           74m
1-amazon-cloud-refresh-18-19-20           0/1       1            0           2d9h
1-amazon-cloud-refresh-84-85-86           0/1       1            0           25h
1-amazon-cloud-refresh-89-90-91           0/1       1            0           25h
1-azure-cloud-event-catcher-59            0/1       1            0           28h
1-azure-cloud-metrics-collector           2/2       2            2           32h
1-azure-cloud-refresh-67-68               1/1       1            1           27h
1-azure-cloud-refresh-69-70               1/1       1            1           27h
1-ems-metrics-processor                   2/2       2            2           41h
1-event-handler                           1/1       1            1           3d
1-generic                                 2/2       2            2           3d
1-google-cloud-event-catcher-16           0/1       1            0           2d9h
1-google-cloud-event-catcher-35           1/1       1            1           2d4h
1-google-cloud-event-catcher-39           1/1       1            1           46h
1-google-cloud-event-catcher-98           1/1       1            1           23h
1-google-cloud-metrics-collector          2/2       2            2           41h
1-google-cloud-refresh-16                 0/1       1            0           2d9h
1-google-cloud-refresh-35                 1/1       1            1           2d4h
1-google-cloud-refresh-39                 1/1       1            1           46h
1-google-cloud-refresh-98                 1/1       1            1           23h
1-google-network-refresh-17               1/1       1            1           2d9h
1-google-network-refresh-36               1/1       1            1           2d4h
1-google-network-refresh-40               1/1       1            1           46h
1-google-network-refresh-99               1/1       1            1           23h
1-openshift-container-event-catcher-150   1/1       1            1           37m
1-openshift-container-metrics-collector   2/2       2            2           37m
1-openshift-container-refresh-150         1/1       1            1           37m
1-priority                                2/2       2            2           3d
1-remote-console                          1/1       1            1           3d
1-reporting                               2/2       2            2           3d
1-schedule                                1/1       1            1           3d
1-ui                                      1/1       1            1           3d
1-web-service                             1/1       1            1           3d

vmdb_production=# select count(id), type, array_agg(queue_name) from miq_workers group by type order by type;
 count |                                   type                                   |                                       array_agg

-------+--------------------------------------------------------------------------+---------------------------------------------------------------------------
------------
     1 | ManageIQ::Providers::Amazon::AgentCoordinatorWorker                      | {ems_0}
     5 | ManageIQ::Providers::Amazon::CloudManager::EventCatcher                  | {ems_2,ems_5,ems_135,ems_144,ems_147}
     2 | ManageIQ::Providers::Amazon::CloudManager::MetricsCollectorWorker        | {amazon,amazon}
     2 | ManageIQ::Providers::Amazon::CloudManager::RefreshWorker                 | {"[\"ems_144\", \"ems_145\", \"ems_146\"]","[\"ems_135\", \"ems_136\", \"e
ms_137\"]"}
     2 | ManageIQ::Providers::Azure::CloudManager::EventCatcher                   | {ems_67,ems_69}
     2 | ManageIQ::Providers::Azure::CloudManager::MetricsCollectorWorker         | {azure,azure}
     2 | ManageIQ::Providers::Azure::CloudManager::RefreshWorker                  | {"[\"ems_67\", \"ems_68\"]","[\"ems_69\", \"ems_70\"]"}
     3 | ManageIQ::Providers::Google::CloudManager::EventCatcher                  | {ems_35,ems_39,ems_98}
     2 | ManageIQ::Providers::Google::CloudManager::MetricsCollectorWorker        | {google,google}
     3 | ManageIQ::Providers::Google::CloudManager::RefreshWorker                 | {ems_39,ems_98,ems_35}
     3 | ManageIQ::Providers::Google::NetworkManager::RefreshWorker               | {ems_36,ems_99,ems_40}
     1 | ManageIQ::Providers::Openshift::ContainerManager::EventCatcher           | {ems_150}
     2 | ManageIQ::Providers::Openshift::ContainerManager::MetricsCollectorWorker | {openshift,openshift}
     1 | ManageIQ::Providers::Openshift::ContainerManager::RefreshWorker          | {ems_150}
     2 | MiqEmsMetricsProcessorWorker                                             | {ems_metrics_processor,ems_metrics_processor}
     1 | MiqEventHandler                                                          | {ems}
     2 | MiqGenericWorker                                                         | {generic,generic}
     2 | MiqPriorityWorker                                                        | {generic,generic}
     1 | MiqRemoteConsoleWorker                                                   | {NULL}
     2 | MiqReportingWorker                                                       | {reporting,reporting}
     1 | MiqScheduleWorker                                                        | {NULL}
     2 | MiqSmartProxyWorker                                                      | {smartproxy,smartproxy}
     1 | MiqUiWorker                                                              | {NULL}
     1 | MiqWebServiceWorker                                                      | {NULL}
(24 rows)

In this case the deployment 1-amazon-cloud-refresh-10-8-9 should not exist as there is no amazon refresh worker with queues ems_8, ems_9, and ems_10.
For this we need to figure out how this deployment didn't get deleted when the worker was removed then either solve that problem or create some generic cleanup function that will make the deployments match what is in the workers table.

Completely unrelated but @agrare why do the RefreshWorker array_agg entries look so different from the rest?  Is it doing some sort of stringified-array?

@Fryguy that is the "child-manager refresher consolidation" so that the CloudManager/NetworkManager/StorageManager are all refreshed by one worker.

do you see log.info("SQL Record for #{w.format_full_log_msg}, Status: [#{w.status}] is being deleted") in the logs?
The other way the pod can remain but the worker row removed is if this exception is raised and they restart the "orchestrator" as I believe that still kills the existing worker rows at startup, right?

@jrafanie I wonder if the worker deletes it's row if it crashes?
ref: https://github.com/ManageIQ/manageiq/blob/master/lib/workers/bin/run_single_worker.rb#L115-L117

haha, a 3rd option.

clean_worker_records logs the message I posted above...
check_not_responding doesn't log but it also shouldn't be in play with the big old return if MiqEnvironment::Command.is_podified? guard clause at the front
run_single_worker.rb as you found also logs the delete when the worker runner exits safely or unsafely.

I think we should get a message in the log if either of the possible options happened.

So I reproduced the generic worker failure by just adding and removing a provider, but the worker deployments did get removed so even though it's definitely a problem, it might be unrelated.

So I think we're going to go with the "create some generic cleanup function that will make the deployments match what is in the workers table" option from the issue description.
How
From a high level we need to be able to correlate the number of workers in kube with the number that we expect (from the database). We can get this info from kube by finding the number of replicas for worker types that support multiple replicas and comparing directly to the deployments for the "deployment per worker" case.
We should label our worker deployments consistently so that they're easy to search for. I think the easiest thing to do would be to label all our deployments with some common info then sort through those deployments in memory. That's similar to what we did in topological-inventory.
Here we get the deployment configs with a generic with a label selector using kubeclient, here we process those returned values for more specific information and here is where we define the labels on creation.
I think this is the way to go because the number of deployments will be relatively small so it doesn't make sense to do a round trip to the OpenShift api for each worker type or something similar.
As for what the labels should be we should have something simple like manageiq/worker=true maybe? That would allow us to pull back all the worker deployments in one call. Then we would need to be able to identify them by worker class name and possibly by ems id (for the ems case). We could do that with the label value, an annotation, or reversing the deployment name. I vote for an annotation here, but any of those would work.
Once we have all the information from kube we probably just want to search for deployments that shouldn't be present rather than also trying to reconcile the number of replicas. I think the logic at this point would be for each deployment to query if we have any workers matching workers in the table.
What
The logic at that point is up for debate. For provider workers, we should be able to determine if we have a matching ems. If we do, the worker stays (it likely is crashing for some valid reason, or just hasn't created its worker record yet). If we don't have a matching provider record it goes (this would mean we're in some strange situation where the worker deployment would never be removed).
I'm not really sure what to do for non-provider workers. I don't think we've ever encountered this issue for that case and I'm not sure that we would be able to tell the difference between something that was just coming up and something that shouldn't be present (there wouldn't be a worker record in either case). We could evaluate the roles and settings and all of that over again, but that seems like overkill. Maybe we start with just doing this for provider workers then expand to others if we need.
When
This could be a new server method called from the monitor loop, but the more I think about the logic, the more it feels like it makes sense right next to the existing worker reconciliation. We're going to have the information by worker type so maybe we should bake it into sync_workers. I think this decision can be made later depending on how the implementation shakes out.

I think this would help the systemd side as well to catch essentially the same issue, service/unit files for workers that shouldn't be running, :+1: sounds good to me

Thinking about this more, it feels like this would be a good first step to removing the workers table (or at least its role in reconciliation). The problem we have here is that we're treating the table as "reality" when really we can inspect what is actually happening in the environment. What the worker manager should be doing long term is comparing what we think we should have (based on roles, servers, zones, settings, providers, etc) and comparing that directly to what exists currently (unit files/pods).
It feels like the worker table is really a crutch because we couldn't check the process space for miq worker processes.


[Tizen] Explicitly depend on the content_browser target.

This is in preparation for M44: tizen_data_fetcher_shared_memory.h includes content/ headers, which in M44 end up depending on blink_headers. Since there was no explicit dependency on content_browser in the xwalk_tizen_lib target, Blink's include paths were not being added to the compiler.

@darktears @mrunalk

LGTM


Solar System object semimajor axes given in km but labeled as au in portal

When exploring the Fink portal for solar system objects, I noticed that the semimajor axes are listed as being given in au, but are actually given in km.
For example, as on this query page: https://fink-portal.org/ZTF23aabfojm, 163693 Atira has the following semimajor axis listed under "dynamical parameters":
a (au): 110850862.8052
Either km or au are useful units for solar system objects, however, au may be the most useful option. In either case, the units listed should be in agreement with the value given.

Thank you @sgreenstreet ! Indeed the current display does not make sense. I will proceed with the required changes (AU).
(also -- you opened the issue on the fink-broker repository, and I transfer it in this repository which contains the code for the portal)

Fixed in https://github.com/astrolabsoftware/fink-science-portal/pull/602 !

Thank you, @JulienPeloton!


feat: support fetchMostRecentBeforeStart time series data request setting

Overview
fetchMostRecentBeforeStart is needed for the line chart to display leading lines and for the status timeline to display pre-viewport status. This PR adds support for this time series data request setting.
Leading line example:

Legal
This project is available under the Apache 2.0 License.

we'll also want to provide this option as default to on for the components where it makes sense, if they aren't all ready.
Are we 100% confident all the caching behavior works as appropriately for this scenario?


Extend Envelop plugin API with the ability to access and replace resolver function

Related: https://github.com/dotansimha/envelop/issues/383

This is brilliant, thank you so much!


Apt repo missing Release file for Focal Fossa

Describe the bug
Installing using the instructions here on Ubuntu 20.04 Focal Fossa:
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest
Results in:
E: The repository 'https://packages.microsoft.com/repos/azure-cli focal Release' does not have a Release file.
To Reproduce
On a clean install of Ubuntu 20.04 Focal Fossa run:
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
Expected behavior
az cli tool will install via apt.

Doh, dupe of #13081 closing.


feat(pixels): add my new pixel

Checklist

[x] I ran npm test locally and it passed without errors.

[x] I only edited the _data/pixels.json file.

[x] I entered the username in the pixels.json that I'm also using to create this pull request.
[x] I acknowledge that all my contributions will be made under the project's license.

Hello sir !
i am new to contribution if something in wrong in my contribution , so please guild me what do i next for you.


Register com.jcraft.jsch.jbcrypt.JBCrypt for reflection

Fix issue #105 (hopefully, not tested) by registering JBCrypt for reflection.

Can you create a test using the snippet you provided in the issue description?
var keyPair = KeyPair.load(new JSch(), privateKeyBytes, null); keyPair.decrypt(passphrase.getBytes(StandardCharsets.UTF_8)):

I'll try. Problem is that code runs in Quarkus and only fails in native mode, not Java mode. Do you have any existing tests that run in native mode, that I can use as a starting point? (I'll look around).

Yes, https://github.com/quarkiverse/quarkus-jsch/blob/dced596cda52584e91f324b58fd33b1ba2a03eae/integration-tests/src/test/java/io/quarkus/it/jsch/NativeJSchIT.java runs in Native.

I also added a test, but your test is nicer. Shall I revert mine?

Sure, I think it's better if we don't store any RSA keys in the repository :)

I haven't been able to actually test that it works. I had Visual Studio installed when this computer crashed and since then I can't reinstall it, so I can't compile native code without doing it in Docker. Can't find a way to run the integration tests in Docker, unfortunately.
I have verified with the real Quarkus release that adding com.jcraft.jsch.jbcrypt.JBCrypt to reflection-config.json helps.

That's great to hear. For some reason my test passes even without your fix, but I'll investigate what's going on

Reverted. As for the RSA key, it is harmless, it was created for these tests, but it is still nicer to do without it.

PR Merged. Thanks! Can you build and try this extension locally and see if that resolves your problem?

I found out how to reproduce the issue in the test, it happens with OpenSSL keys only. I'll update it

@all-contributors add @erik-wramner for code

@gastaldi I can confirm that the new version solves the issue when I build locally.
I have a question though. To make it work, I need to depend on quarkus-jsch-deployment. However, if I follow the installation docs and run mvnw quarkus:add-extension -Dextensions="io.quarkiverse.jsch:quarkus-jsch" the dependency becomes quarkus-jsch. That does not work (even with 999-SNAPSHOT).
Are the installation instructions wrong, so that the correct dependency is quarkus-jsch-deployment, or have I missed something?

Hm, I thought so. Weird - but then perhaps the Maven plugin finds the released version, so I needed to depend on the deployment artifact explicitly to get 999-SNAPSHOT?

No, the resolver should pick from the local repository first. That sounds like a bug, we've been doing some changes in the Maven integration recently and I wonder if this is a new bug.
Can you open an issue in https://github.com/quarkusio/quarkus/issues with a small reproducer project?

I'll see if it is reasonably easy to reproduce. My real project is complex with multiple Maven projects. Stripping it to a small reproducer is not trivial, but if the problem appears in a single Maven project I'll create an issue for it.

@gastaldi I created a single-project reproducer, but it didn't have the problem. It (correctly) uses the local version without having to drag in the deployment project. It could be one of the many other dependencies that creates a conflict or it may be something with the multiple Maven projects. I could keep adding things, trying to make the example project even more like the real one, but given how long time a single build takes and how many variables there are to tweak I'm afraid I don't have the time.

@erik-wramner I've been trying to reproduce your use case and haven't seen it fail. Perhaps you could output the logs with mvn clean package -X? That would give you the -deployment artifacts being used by the Quarkus plugin

I'd prefer not to upload the full trace to the world as it contains a bit too much that may be sensitive. However, I think this is where it goes wrong:
[DEBUG]    com.icecoresoft.connect:filetransfer-lib:jar:2.2-SNAPSHOT:compile
[DEBUG]       io.quarkiverse.jsch:quarkus-jsch:jar:3.0.1:compile (version managed from 999-SNAPSHOT)

For some reason it decides that the version from Quarkus BOM is better than the one in the pom file. I'll move things around a bit to see if I can find out why.

Indeed, plus further down in the trace I get:
[DEBUG] [io.quarkus.bootstrap.resolver.maven.ApplicationDependencyTreeResolver] Injecting deployment dependency io.quarkiverse.jsch:quarkus-jsch-deployment:jar:999-SNAPSHOT
[DEBUG] [io.quarkus.bootstrap.classloading.QuarkusClassLoader] Adding elements io.quarkus.bootstrap.classloading.PathTreeClassPathElement[io.quarkiverse.jsch:quarkus-jsch-deployment / runtime=false resources=null] to QuarkusClassLoader Augmentation Class Loader: PROD

That looks correct. I'm building again with a few changes, perhaps Maven gets confused as the library is pulled in from two sub-projects.

Actually, if you check the bom for 3.2.1 you'll find:
<dependency>
  <groupId>io.quarkiverse.jsch</groupId>
  <artifactId>quarkus-jsch-deployment</artifactId>
  <version>3.0.1</version>
</dependency>

Perhaps you already know, but it's important to mention that Maven resolves to the version that is nearest to the resolved dependency tree. Better use dependencyManagement in the parent pom and declare the version there just in case (although not required, declaring the -deployment there helps)

Actually, if you check the bom for 3.2.1 you'll find:
<dependency>
  <groupId>io.quarkiverse.jsch</groupId>
  <artifactId>quarkus-jsch-deployment</artifactId>
  <version>3.0.1</version>
</dependency>

Oh that's news to me, I was looking at https://github.com/quarkusio/quarkus/blob/main/bom/application/pom.xml but it looks like some preprocessing is run before the artifact is deployed

Well, since the dependency is in the Quarkus BOM, it is already present in the parent's dependencyManagement section. If that is supposed to win, we have found the problem?

Right, so declaring the extension AND the deployment with the new version BEFORE the quarkus-bom should fix the problem


Video sizing

Adds a package json file for future fun and fixes the yt video sizing.

Good stuff, but would it be possible to move the videos to left aligned?

totes!

Addressed

The videos now have a max width instead and are left aligned.


[Proposal] Introduce style enforcement

From my experience it is important (if not vital) to keep code style consistent across the whole codebase. Consistency improves making assumptions about how things should work but also improves readability (and as we know - readability counts).
Currently, the repository has a mandatory black step on travis but this is something that happens in the background during CI and it seems that the black changes are not committed to the main branch (please correct me if I'm wrong here). Locally running black gives me 142 files would be reformatted, 27 files would be left unchanged..
With this PR I would like to introduce enforcing a code style dictated (mostly) by PEP (through flake8) and black with small modifications and hook to pre-commit. This combination offers a quite easy transition and allows developers not to think/remember about formatting their code according to the guidelines.
Small explanations:
flake8 is a great toolkit for checking your code base against coding style (PEP8), programming errors (like “library imported but unused” and “Undefined name”) and to check cyclomatic complexity.
black is a code formatter. In essence it modifies the *.py files to be consistent with the defined rules. In this setup it would do the formatting in the background when commit command is executed.
auto-commit is a library that executes some code when the developer executes git commit command. For this change it means that auto-commit will trigger black and flake8 commands. If any of those commands return something else than 0 ('all good') the person will not be able to commit their changes (It is still possible to commit without style checking by adding --no-verify to the command. ).
After this PR is merged, each person who wants to commit to this repository would have to have precommit installed (possibly also flake8?).  As you might notice, I have not changed any *.py files yet, this is because the checker and autoformatter works lazily - only on the files that will actually be committed. This means that the person who touches the file will be responsible to format it to a correct form if autoformatter is not decisive enough. I am happy to demonstrate this on one of the files if needed.
Example:
I have added one space in tests\tasks\test_multiplex_link_prediction.py file, added this file by git add *.py and executed git commit -m"Test" command.

Since there were failures, the code was not committed. Also, you can see that black modified the source code (files were modified by this hook). Now, it should be enough to manually fix flake8 errors (black ones were fixed automatically) and the commit should be good to go.
At the beginning, this might mean that there will be many false positives because as of now the code does not adhere to PEP8 very well, but because of laziness, we are guaranteed that the change is gradual. However, it is important to introduce consistency as early as possible - the more code there is to fix the worse the experience.
The flake8 and black configurations in this PR are purely my preference, I am happy to change those and/or have a philosophical discussion on whether this is a good configuration. In all projects that I am involved with I am using (almost) exactly this configuration.

Great suggestions! Thank you very much!

I find that the python version is specifically set as 3.6. Are there any ways to support more versions (e.g., 3.6, 3.7, and 3.8)?

@cenyk1230 It is my understanding that setting to 3.6 will support everything 3.6+, however, some 3.7 and 3.8 features might not be checked for.
As far as I could find for black there is no difference between 3.6 and 3.8 because the syntax did not change. I think python did not introduce any big new syntax features since 3.6 (typing) so I think we should be good on that front.

@MaLiN2223 I'm afraid that the auto-commit is not automatically enabled. Could you please help to check whether something needs to be done?

@cenyk1230 Of course, do you have any steps that I can reproduce the issue with?
Alternatively we can also add a git action to reject PR when it does not adhere to the guidelines.

@MaLiN2223 Sorry, it works. But I find that it needs all contributors to run pre-commit install in their machines to enable the pre-commit.

@cenyk1230 that might be the case, if we want to mitigate that issue it would be okay to just autoformat with black and validate with autopep on github action but this makes the feedback loop longer (i.e. author would know that there are some style violations only after the pipeline finished).

@MaLiN2223 Thanks, I have formatted all the code in our repo in #93. Then we should require all the pull requests to follow the code format and style. Are there any quick style check solution that we can use in our travis testing? Maybe we should run flake8 in the before_script of travis.yml?


Read default application.yaml configmap

Coming from spring-cloud-config, and while converting my application into spring-cloud-kuberentes-config.
I found that having a common configuration file for your application help (it can hold database connection configuration, Redis connection, RabbitMQ .... ) so all common configuration can be in one place.
Another missing feature is having config map per profile
For example, if we a spring boot application name  called demo with production profile active then we look for config map with name
application
demo
demo-production
currently only below config map is fetched
demo
Is adding this flexibility on the roadmap, if yes I can start preparing a pull request

I thought it already did that. @iocanel, @geoand or @salaboy care to comment?
If not, it should be considered.

Currently, multiple config maps are supported, but there is no link between Active Profiles and Config Map Names. In case we created multiple config maps per profile they have to be added manually to the spring.application.kubernetes.config.sources.

@shahbour @spencergibb I was also under the impression that we did that.
If not, then I agree it should definitely be considered.
@shahbour Do you have a sample I could try?

ping @shahbour

@spencergibb @geoand , just to be clear we are talking about checking configmaps based on names provided in spring.application.kubernetes.config.sources and active profiles .  If this is the case then yes it is not supported we have to add the configmap name with profile in bootstrap.yml . for example if we have
spring:
  profiles:
    active: production
  application:
    name: cdr-import
  cloud:
    kubernetes:
      config:
        namespace: default
        sources:
         - name: global
         - name: cdr-import

It will only check for configmap global and cdr-import and not global-production and cdr-import-production .
I will provide a sample tomorrow if needed , in my case i had to manually add cdr-improt-produciton to the list

OK, so we do this, but not with *.config.sources is what I am seeing. @geoand agree?

There's an open PR for profiles support https://github.com/spring-cloud/spring-cloud-kubernetes/pull/323

@spencergibb @shahbour yeah, it makes perfect sense

I tried reading PR #323 and I think it is the same as now for reading one config map based on the name provided without postfix any active profile name.
from ConfigMapPropertySource
	private static Map<String, String> getData(KubernetesClient client, String name,
			String namespace, Environment environment) {
		try {
			ConfigMap map = StringUtils.isEmpty(namespace)
					? client.configMaps().withName(name).get()
					: client.configMaps().inNamespace(namespace).withName(name).get();

			if (map != null) {
				return processAllEntries(map.getData(), environment);
			}
		}
		catch (Exception e) {
			LOG.warn("Can't read configMap with name: [" + name + "] in namespace:["
					+ namespace + "]. Ignoring");
		}

		return new HashMap<>();
}

I think we should change the issue name because reading default configuration is already done , what is missing is reading multiple config-map based on active profiles.

@shahbour perhaps open a new issue where you set the appropriate title and link the discussion here?

This issue was opened before providing the spring.application.kubernetes.config.sources which did address partially this issue. Personally, I think the current solution is good unless you think the same behaviour of active profiles and yml files should be mocked into configmap .

I can see reasons for doing both to tell you the truth, so I would like to hear more opinions

I see, ok I have worked on this before in pr #162  , after #323 is merged I can work on it again?

Sounds good to me :)

@geoand @shahbour sorry for my lack of participation here, but I totally agree.. having the profile information while reading configmaps will be really useful

Fixed via 296bf1e


Add BigInt Class

BigInt Class can be used in Competitive Contests which may require the use of BigIntegers to be used in C++

@hegdenaveen1
Can you remove bits/stdc++.h and use the appropriate headers?
In additional, can you use the class keyword instead of structs?