[Istio-1.3.0-rc0] Short protocol detection timeout can fail https requests

Bug description
Follow instructions of Access External Service, everything works till we hit the section access external https service
At this point, after applying service entry for www.google.com, I'm expecting http returns 200 even with registry_only mode, however I still see
kubectl exec -it $SOURCE_POD -c sleep -- curl -I https://www.google.com | grep  "HTTP/"
command terminated with exit code 35

Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[X ] Docs
[ ] Installation
[ X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Steps to reproduce the bug
Version (include the output of istioctl version --remote and kubectl version)
Istio 1.3.0 rc01
How was Istio installed?
GKE v1.12.8-gke.10

@incfly can you double check and make sure this wasn't a silly mistake?

I went through the process again, but still hit the same issue. www.google.com is unable to resolve.

This time I install Istio with helm  template empty options.
I confirmed there's no other Istio SE and VS defined in default namespace to interfere.

I got response when running
$ kubectl exec -it $SOURCE_POD -c sleep -- curl -I https://www.google.com | grep  "HTTP/"
command terminated with exit code 35

Below[1] is the sleep pod listener config snippet. We have two filter chain match, one with prefix_range which throws the request to black hole cluster, the other one is filter chain match on serverName: www.google.com.
The full configuration is attached in gist
@howardjohn does this look different than yours?
[1] 443 port listener section.
{
     "version_info": "2019-09-03T17:58:28Z/23",
     "listener": {
      "name": "<IP_ADDRESS>_443",
      "address": {
       "socket_address": {
        "address": "<IP_ADDRESS>",
        "port_value": 443
       }
      },
      "filter_chains": [
       {
        "filter_chain_match": {
         "prefix_ranges": [
          {
           "address_prefix": "<IP_ADDRESS>",
           "prefix_len": 32
          },
          {
           "address_prefix": "fe80::345a:19ff:fece:378f",
           "prefix_len": 128
          }
         ]
        },
        "filters": [
         {
          "name": "mixer",
          "typed_config": {
           "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
           "transport": {
            "network_fail_policy": {
             "policy": "FAIL_CLOSE",
             "base_retry_wait": "0.080s",
             "max_retry_wait": "1s"
            },
            "check_cluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
            "report_cluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local",
            "report_batch_max_entries": 100,
            "report_batch_max_time": "1s"
           },
           "mixer_attributes": {
            "attributes": {
             "context.proxy_version": {
              "string_value": "1.3.0"
             },
             "context.reporter.kind": {
              "string_value": "outbound"
             },
             "context.reporter.uid": {
              "string_value": "kubernetes://sleep-695c67cb5-b5flj.default"
             },
             "destination.service.host": {
              "string_value": "www.google.com"
             },
             "destination.service.name": {
              "string_value": "www.google.com"
             },
             "destination.service.namespace": {
              "string_value": "default"
             },
             "source.namespace": {
              "string_value": "default"
             },
             "source.uid": {
              "string_value": "kubernetes://sleep-695c67cb5-b5flj.default"
             }
            }
           },
           "disable_check_calls": true
          }
         },
         {
          "name": "envoy.tcp_proxy",
          "typed_config": {
           "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
           "stat_prefix": "BlackHoleCluster",
           "cluster": "BlackHoleCluster"
          }
         }
        ]
       },
       {
        "filter_chain_match": {
         "server_names": [
          "www.google.com"
         ]
        },
        "filters": [
         {
          "name": "mixer",
          "typed_config": {
           "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
           "transport": {
            "network_fail_policy": {
             "policy": "FAIL_CLOSE",
             "base_retry_wait": "0.080s",
             "max_retry_wait": "1s"
            },
            "check_cluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
            "report_cluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local",
            "report_batch_max_entries": 100,
            "report_batch_max_time": "1s"
           },
           "mixer_attributes": {
            "attributes": {
             "context.proxy_version": {
              "string_value": "1.3.0"
             },
             "context.reporter.kind": {
              "string_value": "outbound"
             },
             "context.reporter.uid": {
              "string_value": "kubernetes://sleep-695c67cb5-b5flj.default"
             },
             "destination.service.host": {
              "string_value": "www.google.com"
             },
             "destination.service.name": {
              "string_value": "www.google.com"
             },
             "destination.service.namespace": {
              "string_value": "default"
             },
             "source.namespace": {
              "string_value": "default"
             },
             "source.uid": {
              "string_value": "kubernetes://sleep-695c67cb5-b5flj.default"
             }
            }
           },
           "disable_check_calls": true
          }
         },
         {
          "name": "envoy.tcp_proxy",
          "typed_config": {
           "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
           "stat_prefix": "outbound|443||www.google.com",
           "cluster": "outbound|443||www.google.com"
          }
         }
        ]
       }
      ],
      "deprecated_v1": {
       "bind_to_port": false
      },
      "listener_filters": [
       {
        "name": "envoy.listener.tls_inspector"
       }
      ],
      "listener_filters_timeout": "0.010s",
      "traffic_direction": "OUTBOUND",
      "continue_on_listener_filters_timeout": true
     },
     "last_updated": "2019-09-03T17:58:28.860Z"
    }
   ]
  },

My config (works fine)
[
    {
        "name": "<IP_ADDRESS>_443",
        "address": {
            "socketAddress": {
                "address": "<IP_ADDRESS>",
                "portValue": 443
            }
        },
        "filterChains": [
            {
                "filterChainMatch": {
                    "prefixRanges": [
                        {
                            "addressPrefix": "<IP_ADDRESS>",
                            "prefixLen": 32
                        },
                        {
                            "addressPrefix": "fe80::d843:24ff:fe9e:b85a",
                            "prefixLen": 128
                        }
                    ]
                },
                "filters": [
                    {
                        "name": "mixer",
                        "typedConfig": {
                            "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                            "transport": {
                                "networkFailPolicy": {
                                    "policy": "FAIL_CLOSE",
                                    "baseRetryWait": "0.080s",
                                    "maxRetryWait": "1s"
                                },
                                "checkCluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
                                "reportCluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local",
                                "reportBatchMaxEntries": 100,
                                "reportBatchMaxTime": "1s"
                            },
                            "mixerAttributes": {
                                "attributes": {
                                    "context.proxy_version": {
                                        "stringValue": "1.3.0"
                                    },
                                    "context.reporter.kind": {
                                        "stringValue": "outbound"
                                    },
                                    "context.reporter.uid": {
                                        "stringValue": "kubernetes://shell-684f7478b5-kb4s6.default"
                                    },
                                    "destination.service.host": {
                                        "stringValue": "www.google.com"
                                    },
                                    "destination.service.name": {
                                        "stringValue": "www.google.com"
                                    },
                                    "destination.service.namespace": {
                                        "stringValue": "default"
                                    },
                                    "source.namespace": {
                                        "stringValue": "default"
                                    },
                                    "source.uid": {
                                        "stringValue": "kubernetes://shell-684f7478b5-kb4s6.default"
                                    }
                                }
                            },
                            "disableCheckCalls": true
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "BlackHoleCluster",
                            "cluster": "BlackHoleCluster"
                        }
                    }
                ]
            },
            {
                "filterChainMatch": {
                    "serverNames": [
                        "www.google.com"
                    ]
                },
                "filters": [
                    {
                        "name": "mixer",
                        "typedConfig": {
                            "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                            "transport": {
                                "networkFailPolicy": {
                                    "policy": "FAIL_CLOSE",
                                    "baseRetryWait": "0.080s",
                                    "maxRetryWait": "1s"
                                },
                                "checkCluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
                                "reportCluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local",
                                "reportBatchMaxEntries": 100,
                                "reportBatchMaxTime": "1s"
                            },
                            "mixerAttributes": {
                                "attributes": {
                                    "context.proxy_version": {
                                        "stringValue": "1.3.0"
                                    },
                                    "context.reporter.kind": {
                                        "stringValue": "outbound"
                                    },
                                    "context.reporter.uid": {
                                        "stringValue": "kubernetes://shell-684f7478b5-kb4s6.default"
                                    },
                                    "destination.service.host": {
                                        "stringValue": "www.google.com"
                                    },
                                    "destination.service.name": {
                                        "stringValue": "www.google.com"
                                    },
                                    "destination.service.namespace": {
                                        "stringValue": "default"
                                    },
                                    "source.namespace": {
                                        "stringValue": "default"
                                    },
                                    "source.uid": {
                                        "stringValue": "kubernetes://shell-684f7478b5-kb4s6.default"
                                    }
                                }
                            },
                            "disableCheckCalls": true
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "outbound|443||www.google.com",
                            "cluster": "outbound|443||www.google.com",
                            "accessLog": [
                                {
                                    "name": "envoy.file_access_log",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                        "path": "/dev/stdout",
                                        "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
                                    }
                                }
                            ]
                        }
                    }
                ]
            },
            {
                "filterChainMatch": {},
                "filters": [
                    {
                        "name": "mixer",
                        "typedConfig": {
                            "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                            "transport": {
                                "networkFailPolicy": {
                                    "policy": "FAIL_CLOSE",
                                    "baseRetryWait": "0.080s",
                                    "maxRetryWait": "1s"
                                },
                                "checkCluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
                                "reportCluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local",
                                "reportBatchMaxEntries": 100,
                                "reportBatchMaxTime": "1s"
                            },
                            "mixerAttributes": {
                                "attributes": {
                                    "context.proxy_version": {
                                        "stringValue": "1.3.0"
                                    },
                                    "context.reporter.kind": {
                                        "stringValue": "outbound"
                                    },
                                    "context.reporter.uid": {
                                        "stringValue": "kubernetes://shell-684f7478b5-kb4s6.default"
                                    },
                                    "destination.service.host": {
                                        "stringValue": "www.google.com"
                                    },
                                    "destination.service.name": {
                                        "stringValue": "www.google.com"
                                    },
                                    "destination.service.namespace": {
                                        "stringValue": "default"
                                    },
                                    "source.namespace": {
                                        "stringValue": "default"
                                    },
                                    "source.uid": {
                                        "stringValue": "kubernetes://shell-684f7478b5-kb4s6.default"
                                    }
                                }
                            },
                            "disableCheckCalls": true
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "PassthroughCluster",
                            "cluster": "PassthroughCluster"
                        }
                    }
                ]
            }
        ],
        "deprecatedV1": {
            "bindToPort": false
        },
        "listenerFilters": [
            {
                "name": "envoy.listener.tls_inspector"
            }
        ],
        "listenerFiltersTimeout": "0.010s",
        "continueOnListenerFiltersTimeout": true,
        "trafficDirection": "OUTBOUND"
    }
]

Interesting.. I went on @incfly cluster and found that when applying the SE, it works for like a second, then back to failing.. will investigate more tomorrow

ok this is really weird.. when I run curl -v it fails and when I run curl it works?
If I run httpie it works ~60% of the time.
/ $ for i in {0..10}; do curl -sv https://www.google.com/get; echo $i; done
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
0
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
1
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
2
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
3
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
4
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
5
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
6
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
7
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
8
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
9
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to www.google.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
*  start date: Aug 13 16:20:25 2019 GMT
*  expire date: Nov 11 16:20:25 2019 GMT
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e8c3777900)
> GET /get HTTP/2
> Host: www.google.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< content-type: text/html; charset=UTF-8
< x-content-type-options: nosniff
< date: Wed, 04 Sep 2019 02:59:41 GMT
< server: sffe
< content-length: 1564
< x-xss-protection: 0
<
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/get</code> was not found on this server.  <ins>That’s all we know.</ins>
* Connection #0 to host www.google.com left intact
10

/ $ for i in {0..10}; do curl -s https://www.google.com/get -o /dev/null; echo $i; done
0
1
2
3
4
5
6
7
8
9
10

access log for failed and successful requests look like
"- - -" 0 - "-" "-" 547 4518 8 - "-" "-" "-" "-" "<IP_ADDRESS>:443" outbound|443||www.google.com <IP_ADDRESS>:38654 <IP_ADDRESS>:443 <IP_ADDRESS>:55082 www.google.com -

They always show 0, not 200. Maybe this is expected for https

did you turned on access log somehow? Is that a mesh config field?
one thing I noticed unusual is, for httpbin.org, even though service_entry creation makes a difference, but I can't see the access log entries in sidecar about request.

yes I turned them on via mesh config. I switched  the service entry from
google.com to httpbin.org to make sure it wasn't some google thing and
seems to have same behavior
On Tue, Sep 3, 2019 at 8:26 PM Jianfei Hu<EMAIL_ADDRESS>wrote:

did you turned on access log somehow? Is that a mesh config field?
one thing I noticed unusual is, for httpbin.org, even though
service_entry creation makes a difference, but I can't see the access log
entries in sidecar about request.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/16581?email_source=notifications&email_token=AAEYGXMNSJ4NSZCFKYLSUH3QH4TG3A5CNFSM4IQIIFSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52GVUQ#issuecomment-527723218,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEYGXP2TAFPII2MKW6QRV3QH4TG3ANCNFSM4IQIIFSA
.

I recalled that I once tried cilium, maybe in this cluster. I do delete the deployments and no longer see the cilium pods in kube-system ns, but this might have some left over effect...
I'll create another cluster with same GKE version to see if this can be reproduced...

@rlenglet ^

Cilium can clean up after itself when the daemonset is deleted:
install/kubernetes/cilium/charts/agent/templates/daemonset.yaml
But that's done only when the CILIUM_FLANNEL_UNINSTALL_ON_EXIT environment variable is set to true:
https://github.com/cilium/cilium/blob/8ce482fd82d996f01099803db4f3ea673901172b/plugins/cilium-cni/cni-uninstall.sh#L37
And CILIUM_FLANNEL_UNINSTALL_ON_EXIT  is set to false by default:
https://github.com/cilium/cilium/blob/8ce482fd82d996f01099803db4f3ea673901172b/install/kubernetes/cilium/charts/agent/templates/daemonset.yaml#L52-L57
https://github.com/cilium/cilium/blob/8ce482fd82d996f01099803db4f3ea673901172b/install/kubernetes/cilium/charts/config/templates/configmap.yaml#L174
https://github.com/cilium/cilium/blob/8ce482fd82d996f01099803db4f3ea673901172b/install/kubernetes/cilium/values.yaml#L234
So yeah it's possible that you had a leftover networkpolicy applied by Cilium's BPF programs even though you uninstalled the Cilium daemonset.

Thanks for the pointer @rlenglet but I just created another fresh new cluster and findings are similar to what @howardjohn described. So culprit is not potential cilium leftover.

When mesh is in ALLOW_ANY mode, no ServiceEntry defined, curl consistently succeed, regardless whether -i -v params are passed to curl
When mesh is in REGISTRY_ONLY mode, if ServiceEntry for Google.com is configured,

if we do curl -i -v, it's constantly failing as command terminated with exit code 35. no success.

if curl -i, the pattern is seeing a HTTP 200, then a couple requests fails in a row. This pattern repeats. OK, FAIL, FAIL, FAIL, OK, FAIL, OK...

@howardjohn I'm wondering when you say it works on your cluster, do you see curl -I https://www.google.com constantly returns 200 or it's a one off success?

I didn't pass -I but when I pass -v it fails sometimes and without -v it
works every time
On Tue, Sep 3, 2019, 9:03 PM Jianfei Hu<EMAIL_ADDRESS>wrote:

Thanks for the pointer @rlenglet https://github.com/rlenglet but I just
created another fresh new cluster and findings are similar to what
@howardjohn https://github.com/howardjohn described. So culprit is not
potential cilium leftover.

When mesh is in ALLOW_ANY mode, no ServiceEntry defined, curl
consistently succeed, regardless whether -i -v params are passed to
curl
When mesh is in REGISTRY_ONLY mode, if ServiceEntry for Google.com
is configured,
1.
1. if we do curl -i -v, it's constantly failing as command
terminated with exit code 35. no success.
2. if curl -i, the pattern is seeing a HTTP 200, then a couple
requests fails in a row. This pattern repeats. OK, FAIL, FAIL,
FAIL, OK, FAIL, OK...

@howardjohn https://github.com/howardjohn I'm wondering when you say it
works on your cluster, do you see curl -I https://www.google.com
constantly returns 200 or it's a one off success?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/16581?email_source=notifications&email_token=AAEYGXMVR2JGAPPW6CMTTZLQH4XQVA5CNFSM4IQIIFSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52IK2A#issuecomment-527730024,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEYGXPLOLA2HATSVTVU5TDQH4XQVANCNFSM4IQIIFSA
.

Is this only on GKE?   -v does not change curl behavior.

I guess -v changes the timing of messages due to the verbose printing into stdout.
Could it be caused by protocol sniffing?

The default protocol detection timeout is 10ms: https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/values.yaml#L217
@incfly Could you please try setting it to a much higher value, e.g. 10s? I'd like to rule this out first since it's a recent feature, and the only short timeout I'm aware of in Envoy.

that is a great point. we aren't doing protocol sniffing, but we did add a
new setting that makes the tls inspector time out after 10ms. We must be
too slow, we should bump up the time
On Tue, Sep 3, 2019, 9:18 PM Romain Lenglet<EMAIL_ADDRESS>wrote:

The default protocol detection timeout is 10ms:
https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/values.yaml#L217
@incfly https://github.com/incfly Could you please try setting it to a
much higher value, e.g. 10s? I'd like to rule this out first since it's a
recent feature, and the only short timeout I'm aware of in Envoy.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/16581?email_source=notifications&email_token=AAEYGXPPQI6ODQCATRKZEFLQH4ZIHA5CNFSM4IQIIFSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52I6VQ#issuecomment-527732566,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEYGXOGCV3EIIC6BWUJMUTQH4ZIHANCNFSM4IQIIFSA
.

Is there an easy way to tell from the istio-proxy logs? If this is the issue, I bet it will happen frequently to users, and we should have a quick way of diagnosing this.

I had debug logging on and I didn't notice anything, but maybe I was not
looking for the right thing.
This whole timeout is kind of concerning, that a slow request will be
broken. How long do we set it? No matter what users will hit the timeout at
some point
On Tue, Sep 3, 2019, 9:23 PM Romain Lenglet<EMAIL_ADDRESS>wrote:

Is there an easy way to tell from the istio-proxy logs? If this is the
issue, I bet it will happen frequently to users, and we should have a quick
way of diagnosing this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/16581?email_source=notifications&email_token=AAEYGXMTT5LZM47CKDIS44DQH4Z3DA5CNFSM4IQIIFSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52JF4Q#issuecomment-527733490,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEYGXPHNVTQ2S3H5PIQNLLQH4Z3DANCNFSM4IQIIFSA
.

@rlenglet changing to 10s does make a difference, now curl -i -V reliably succeeds. Great sniffing on the protocol sniffing's problem!
+1 on the tricky to set the timeout...

/cc @incfly

To be clear, the timeout is not only for http inspector, it's for all listener filters. If one of the listener filters times out (in this case, it's very likely to be tls inspector), the default filter chain will be used.
@rshriram 10ms seems to be too short. I am going to increase the timeout to 150ms. It could possibly affect MySQL. WDYT?

@yxue that means that Envoy will wait up to 150ms to receive data from the downstream, but will abort as soon as any data is received?
10ms is a long time between the connect and sending data.  I'm surprised that's not sufficient.
150ms could be a DoS vector.   You could trick Envoy into opening tons of sockets and keeping them open until all ports are exhausted.
I wonder if we should keep it at 10ms but make it easier to debug this.  How could we make debugging this easier?
@rlenglet seriously nice debugging

For release-1.3, the best we can do is set this timeout to a relatively high value to avoid most timeouts. What are the side effects of using a high value? I doubt this increases resource consumption, right?
Post-1.3.0-release, could we think of a design that doesn't rely on a timeout? Why do we need to time out in the first place?

10ms is a long time between the connect and sending data. I'm surprised that's not sufficient.

With curl -v, there may be 1+ roundtrips between the GKE cluster and the client's shell to print out debug messages. I could see that add up to more than 10ms.

this feature original had no timeout, I think it was added so things like
MySQL work which are server first protocols
On Tue, Sep 3, 2019, 10:03 PM Romain Lenglet<EMAIL_ADDRESS>wrote:

For release-1.3, the best we can do is set this timeout to a relatively
high value to avoid most timeouts. What are the side effects of using a
high value? I doubt this increases resource consumption, right?
Post-1.3.0-release, could we think of a design that doesn't rely on a
timeout? Why do we need to time out in the first place?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/16581?email_source=notifications&email_token=AAEYGXL2G76BAY4DR6UPS33QH46TTA5CNFSM4IQIIFSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52LAII#issuecomment-527740961,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEYGXLZQ4XUFRXFA2RBLEDQH46TTANCNFSM4IQIIFSA
.

is the feature without timeout in 1.2 or earlier?

10ms is a long time between the connect and sending data. I'm surprised that's not sufficient.

With curl -v, there may be 1+ roundtrips between the GKE cluster and the client's shell to print out debug messages. I could see that add up to more than 10ms. Right now I see >160ms latency between US West Coast and Europe. 10ms is very small.

Is this used for ingress gateways too or only sidecars?

There is no timeout in 1.2 and earlier.
It's introduced by these two PRs:
Envoy PR: https://github.com/envoyproxy/envoy/pull/7859
Istio PR: https://github.com/istio/istio/pull/16223

@yxue that means that Envoy will wait up to 150ms to receive data from the downstream, but will abort as soon as any data is received?

150ms is the timeout from for listener filters. Envoy will wait up to 150ms for listener filters to inspect the packet in Istio's case.

Envoy will wait up to 150ms to receive data from the downstream, but will abort as soon as any data is received?

150ms is the timeout from for listener filters. Envoy will wait up to 150ms for listener filters to inspect the packet in Istio's case. Even though data is received, Envoy will still for listener filters.

Is there a way to disable this/is it only enabled by some special config or can we only increase the timeout?    I think there needs to be a way of disabling it (e.g,. if 0ms don't use protocol detection)
When it is enabled, I think it needs to be easy to debug so we and our users are not always chasing intermittent needle-in-a-haystack timeouts.
Is there a design doc perhaps for this?

design doc:  https://docs.google.com/document/d/12Z_oFCFn3_rTWbQsuR1GmjU5_WZVkBTJwFpFMx802Ns/edit#heading=h.t9r3j7wzaz04

The downside of a large timeout is the server protocol will experience the timeout at each sidecar.

Debugability: each inspector (tls and http) exports counter of failure reason see https://github.com/envoyproxy/envoy/blob/master/source/extensions/filters/listener/tls_inspector/tls_inspector.h#L21-L30
Wait, the counter is not bump if it's the listener decide to bypass the listener filter. Let me see if there is any log/counter binding to listener timeout...

Nope. Will add one counter

I think we can use this counter https://github.com/envoyproxy/envoy/blob/d504fde0ffd97017d1ddff8caa9a3b46bba9ae48/source/server/connection_handler_impl.cc#L159

my 2c -
With no timeout (infinite), we either succeed 100% (https) or 0% (MySQL). if you fail all the time you know you have a problem and we can make it easily to debug and fix.
If we add a timeout, your https requests will fail say 1% of the time, which is hard to debug, and your MySQL requests may work but they are really slow, which is both hard to detect and debug. Additionally, a timeout makes downgrade attacks trivial.

I have the same issue, requests to wikipedia.org fail intermittently.

@louiscryan @howardjohn and @yxue and I had a chat.  The plan for 1.3.0 is:

@howardjohn will disable tls inspector for ingress gateways (or if it can't be removed, ensure that the listener filter timeout will not affect it).  It will only be enabled for sidecars and egress gateways.
@yxue will increase the default timeout to 1 sec
@yxue will add a DEBUG-level log message to Envoy whenever the timeout is hit
best effort/if there is time (and there probably isn't, but it can go into 1.3.1):  @yxue will export timeout metrics from Envoy through Istio.

tls_inspector on gateways is from AUTO_PASSTHROUGH
reproduce:
cat <<EOF | kaf -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: my-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "foo.bar"
    port:
      name: tls
      number: 15678
      protocol: TLS
    tls:
      mode: AUTO_PASSTHROUGH
EOF

sorry I am late to the party. tlsInspector isnot used in the gateways. If it is, then we should remove it. and on Sidecars, we need the 10ms timeout in the ingress listener (not egress) and on the egress listener, we could have a longer timeout like 500ms utmost.
Now, the bigger question is, on a sidecar, why was 10ms not sufficient for the HTTP protocol sniffer to detect that this was not HTTP traffic? The client (from sidecar envoy's perspective) is the localhost user process. So, surely, the TLS blocks from https://google.com should have arrived at the sidecar in a jiffy, with plenty of data for the http_inspector to detect that this is NOT HTTP traffic.

Also, this issue could become a non issue with the recent default enabling of tls_inspector in all listeners with http_inspector

@rshriram

tlsInspector isnot used in the gateways. If it is, then we should remove it.

Maybe I miss something, tls inspector is used when there is a filter chain selection by SNI. So when TLS-by-SNI routing is performed (by the gateways or sidecars), tls inspector is there.
https://www.envoyproxy.io/docs/envoy/v1.8.0/configuration/listener_filters/tls_inspector

TLS Inspector listener filter allows detecting whether the transport appears to be TLS or plaintext, and if it is TLS, it detects the Server Name Indication ... from the client. This can be used to select a FilterChain via the server_names

Now, the bigger question is, on a sidecar, why was 10ms not sufficient for the HTTP protocol sniffer to detect that this was not HTTP traffic?

The hypothesis is that running curl with -v slowed it enough due to remote shell output.

I also find this: client may do a slow dns reverse lookup between tcp handshake and ssl handshake. Any queuing at client and envoy wastes the ticks.
https://unix.stackexchange.com/questions/128370/linux-openssl-cn-hostname-verification-against-ssl-certificate

Wouldn't it do that after sending the ClientHello and receiving the ServerHello?

Yeah I was about to ask the same.. and in this case, the http_inspector or tls_inspector would have completed its function

Copy pasting my response from the PR comment.
So here is a possible solution:

Set the listener filter timeout on all ingress listeners for sidecars with TLS_inspector/http inspector and multiple filter chains (i.e. permissive mode).
set the listener filter timeout on all egress listeners on sidecars with http_inspector
In both cases, you do not have to increase the default timeout. Or may be we can set it to 100ms on the ingress listener. On the egress listener, it should always be small (10ms) because the traffic is coming from localhost loopback socket and not over the wire.
In all other cases, do not set any timeout.

Now, what is perplexing is this hypothesis

The hypothesis is that running curl with -v slowed it enough due to remote shell output.

Conclude the internal discussion:

Setting timeout for inbound listener to 1s. The reason is considering the RTT in network, e.g., RTT for cross continent traffic
Setting timeout for outbound listener to 100ms. Traffic is from localhost.

cc @duderino @costinm @rlenglet  @howardjohn
Copy past proposed solution from @rshriram
when we construct the filter chains, we add some metadata or some other param to track how many protocols are being detected..

if its http & tcp only, then set 10ms timeout
if its https only (i.e. sni based routing), no timeout
if its http, https, and tcp, then it is a case where the user has both http and https services on same port. We can set a large timeout here like 200ms.
on inbound, we should not increase timeout at all IMO. inbound scenario is straight forward. its always between http and tcp [with or without tls sniffing]. or if we feel too nervous about 10ms, then we can bump it to 25ms. But really, the 90% of k8s clusters are inside one dc where RTTs are <1ms

Setting timeout for inbound listener to 1s. The reason is considering the RTT in network, e.g., RTT for cross continent traffic
Setting timeout for outbound listener to 100ms. Traffic is from localhost.

LGTM.

Setting 1s and 100ms for inbound and outbound is a trade-off considering it's near release-1.3. We did hit the problem when setting 10ms for outbound and don't have enough time to benchmark the correct value here.
cc @louiscryan @rshriram

This is still marked as release blocker, we fixed by raising timeouts right?

We're experiencing something very similar in one of our clusters (in AWS Australia). We've noticed two issues in that region.
First: external calls fails intermittently only requests go through the sidecar. If we exclude the calls by port or external destination IP, the calls always succeed.
Second: All of our elasticsearch nodes running in the cluster fail to talk to each other intermittently forcing the clusters to go into yellow state.
The two issues don't seem to be related but keep happening only in one cluster (out of 5 so far) and only when the affected services have sidecars added. Can someone please clarify what how this issue was resolved eventually?

The protocol detection timeout was removed (set to infinite)

Thank you. I understand this shouldn't cause issues with slow external calls anymore but will break server first protocol services as long as they don't specify appProtocol for their ports, right? If appProtocol is specified then even server-first services should work. How wrong am I? :)

Correct. The decision was that its better to fail hard and make someone go solve the issue explicitly than to either make all requests hang for ~5s and/or have flaky connections  which is harder to track down.

That makes total sense. Unfortunately for me it means my issues lie elsewhere hah :) I guess I'll continue debugging


node-ipc

Adds definition for npm module node-ipc

node-ipc/node-ipc.d.ts
check list

[ ] is collect naming convention?

https://www.npmjs.com/package/node-ipc
http://bower.io/search/?q=node-ipc
others?

[X] has a test file? (node-ipc/node-ipc-tests.ts or others)
[ ] pass the Travic-CI test?

@adamringhede thank you for your contributions.
please use declare module "node-ipc", like this https://github.com/borisyankov/DefinitelyTyped/blob/master/rimraf/rimraf.d.ts#L8.

Hello and thank you for your contribution!
Due to an excessively long queue of pull requests that have become stale, we are "declaring bankruptcy" and closing all PRs opened before May 1, 2016. If you'd still like to merge this code in, please open a new PR that has been merged and rebased with the master branch.
Going forward, we are committing to review or merge all PRs on a regular basis so this bankruptcy will not occur again. We apologize for the incovenience and hope you will continue to contribute to DefinitelyTyped in the future.
Thanks
The TypeScript and DefinitelyTyped teams


[Bug] Template switches not being matched

Hi, I tried adding template switches to the hub configuration but somehow they don't appear. The following is the hub configuration:
"include": [
      {
        "type": "pattern",
        "value": "switch.all_alexa"
      },
      {
        "type": "label",
        "value": "all_alexa"
      },
      {
        "type": "domain",
        "value": "switch"
      },
    ]

As you can see, I tried including all kinds of filters, but the only switches found are the one that are not templates.
BTW thank you very much for your work 😃

Sorry, forgot to add that I'm using the docker image and not the addon. In any case, I've tried restarting the container but there are still no template switches

In general it doesn't make any difference, where a device is coming from - if its any integration or a template.
Anyway i just validated that template switches in general work.
Maybe you can share a bit more details:

number of bridges
full configuration of the bridge
screenshot of the bridge and its devices
log while creating the bridge
attributes of at least one missing template switch (HA -> Developer Tools -> State -> search for the entity -> Screenshot)

Found the problem! I figured that the only difference between my template switches (manually created in YAML file) and other switches, is that they do not have an unique ID, which is not mandatory.
After adding one in the unique_id field, they now appear in the bridge devices.
I'm guessing that somewhere in HASS APIs or in your code, entities without an unique ID get filtered out


No shortcut when using "requested-execution-level": "requireAdministrator"

I'm using forge and I need to request admin rights for my app but when i do that, there no shortcut created on the Desktop. When i remove requested-execution-level from my package.json, it works.
What should I do ?
"forge": {
      "packagerConfig": {
        "name": "MyApp",
        "win32metadata": {
          "requested-execution-level": "requireAdministrator"
        }
      },

Same here. Pretty wired.

I just resolve this problem by handling squirrel event: --squirrel-firstrun, based on project electron-squirrel-startup
You can check my code below. In my case I just quit immediately after create shortcut on firstrun.
`
var path = require('path');
var spawn = require('child_process').spawn;
var app = require('electron').app;
var run = function(args, done) {
var updateExe = path.resolve(path.dirname(process.execPath), '..', 'Update.exe');
// debug('Spawning %s with args %s', updateExe, args);
spawn(updateExe, args, {
detached: true
}).on('close', done);
};
var check = function() {
if (process.platform === 'win32') {
var cmd = process.argv[1];
// debug('processing squirrel command %s', cmd);
var target = path.basename(process.execPath);
if (cmd === '--squirrel-firstrun') {
  run(['--createShortcut=' + target + ''], app.quit);
  return true;
}

if (cmd === '--squirrel-install' || cmd === '--squirrel-updated') {
  run(['--createShortcut=' + target + ''], app.quit);
  return true;
}
if (cmd === '--squirrel-uninstall') {
  run(['--removeShortcut=' + target + ''], app.quit);
  return true;
}
if (cmd === '--squirrel-obsolete') {
  app.quit();
  return true;
}

}
return false;
};
module.exports = check();
`

Hey thanks for the hint
I'd like to do the same, but if my app declares "requested-execution-level": "requireAdministrator"
No Squirel hook is being called (so I don't get a --squirrel-uninstall for ex)
Did you found a solution for this ?

Same here, any update?

Same here, any update?

I am also facing the same issue, and it seems there is no solution yet. :(

Same here, any update?

I am also facing the same issue, and it seems there is no solution yet.


Podman won't install on Fedora 31 after changing cgroup to v1

On a Fedora 31 machine, I installed Docker with dnf -y install moby-engine and then followed this procedure to set cgroup from the default v2 to v1:
As Root:
# vi /etc/default/grub
Add 'systemd.unified_cgroup_hierarchy=0' to the GRUB_CMDLINE_LINUX line
# grub2-mkconfig
# shutdown –r now

Then after creating a few containers using Docker after the reboot, I installed podman and received this:
# dnf -y install podman
Fedora Modular 31 - x86_64                                                                        11 kB/s | 3.4 kB     00:00
Failed to download metadata for repo 'fedora-modular'
Error: Failed to download metadata for repo 'fedora-modular'

It was installing latest Podman, I believe v1.6.2.

I don't think the error is related to the version of cgroup you are using.
Could you try running dnf with --verbose?


If I want to use the the meta-parameters to adapt to new task, what should I do?

I write a new environment (navigation on deterministic map):
(1) I run " python train.py --config xxxx", and get config.json, policy.th.
(2) I run "python test.py -config xxxx", and get results.npz.
But the rewards in results.npz are still very low.
What should I do to use policy.th to fast adapt to a new task?

You should use --policy policy.th in test.py to use your trained policy.
That's surprising that you didn't get any error when running test.py without --policy, since this is a required parameter.

I get it. I run test.py with policy.th. But the rewards of valid_return are equal to or even lower than train_return.
Maybe, our environment is not suitable. Thanks.


ARTEMIS-1324 Deadlock detection and health check of critical components

What is the over head of this, latency and throughput?

I will make some changes here tomorrow.

@clebertsuconic I've left more code nits, and user feedback around the docs.
Not really able to put the code through its paces, no doubt won't be able to do that till it hits a real environment, it seems reasonable the pseudo logic you describe.
Also the test cases added look like they cover some good scenarios.


[$1000] Copying code blocks from chat is broken

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Navigate to any chat
Copy any code block
paste in chat

Expected Result:
User should be able to copy and paste code blocks
Actual Result:
What ends up being copied to your clipboard is not the content that is highlighted
Workaround:
unknown
Platform:

Where is this issue occurring?

Web
iOS
Android
Desktop App
Mobile Web

Version Number: 1.1.37-0
Reproducible in staging?: Y
Reproducible in production?:  Y
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://user-images.githubusercontent.com/43995119/153130275-fe25f652-8fc5-439d-b034-58934578ad50.mp4
Expensify/Expensify Issue URL:
Issue reported by:  @puneetlath
Slack conversation: https://expensify.slack.com/archives/C01GTK53T8Q/p1643914668204029
Job Post https://www.upwork.com/jobs/~0103d56637d4154cbd
View all open jobs on GitHub

Triggered auto assignment to @aldo-expensify (Engineering), see https://stackoverflow.com/c/expensify/questions/4319 for more details.

Triggered auto assignment to Contributor-plus team member for initial proposal review - @rushatgabhane (Exported)

Triggered auto assignment to @iwiznia (Exported), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

👋 @ihamzanizami , I suggest you read https://github.com/Expensify/App/blob/main/CONTRIBUTING.md to learn how contributors can participate and propose solutions.

I understand that the problem are the line breaks. When I copy code blocks, I always have to fix line breaks manually.

Oh, I see. FWIW it works properly if you CMD+SHIFT+V (paste without format)

Proposal
issue happend on htmlconverter .  since the code block is formated with html .
code formate use
 as a new line and we net parse this on library.
we can fix this by just parsing the div as new line  /n or
 and html converter will count it as a new line.
in
https://github.com/Expensify/App/blob/403fc28984e9d96f3127257a3dc3ac6e66a75cd6/src/components/TextInputFocusable/index.js#L244
+ const fixHtml = html.replaceAll('
', '').replaceAll('
', '
');
- const markdownText = parser.htmlToMarkdown(html);
+ const markdownText = parser.htmlToMarkdown(fixHtml);

we can fix this from the library also by edit this.
            {
                name: 'newline',

                // Replaces open and closing

 tags with a single

                // Slack uses special  tag for empty lines instead of
 tag
                pre: inputString => inputString.replace('

', '
').replace('

', '
')
                    .replace(SLACK_SPAN_NEW_LINE_TAG + SLACK_SPAN_NEW_LINE_TAG, '

').replace(SLACK_SPAN_NEW_LINE_TAG, '

'),

                // Include the immediately followed newline as `<br>\n` should be equal to one \n.
                regex: /
<])*>\n?/gi,
                replacement: '\n'
            },

                pre: inputString => inputString .replaceAll('
', '').replaceAll('
', '
')replace('

', '
').replace('

', '
')
                    .replace(SLACK_SPAN_NEW_LINE_TAG + SLACK_SPAN_NEW_LINE_TAG, '

').replace(SLACK_SPAN_NEW_LINE_TAG, '

'),

            pre: inputString => inputString .replaceAll('
', '').replaceAll('
', '
')replace('

', '
').replace('

', '
')
                .replace(SLACK_SPAN_NEW_LINE_TAG + SLACK_SPAN_NEW_LINE_TAG, '

').replace(SLACK_SPAN_NEW_LINE_TAG, '

'),

@ahmdshrif that didn't work for me.
I modified the new line rule of htmlToMarkdownRules

We just modified the pre with this  not all object

you can apply the n /node_modules/expensify-common/lib/ExpensiMark.js
line 197
 {
                name: 'newline',

                // Replaces open and closing

 tags with a single

                // Slack uses special  tag for empty lines instead of
 tag
-                pre: inputString => inputString.replace('

', '
').replace('

', '
')
+                pre: inputString => inputString .replaceAll('
', '').replaceAll('
', '
').replace('

', '
').replace('

', '
')
                .replace(SLACK_SPAN_NEW_LINE_TAG + SLACK_SPAN_NEW_LINE_TAG, '

').replace(SLACK_SPAN_NEW_LINE_TAG, '

'),

                // Include the immediately followed newline as `<br>\n` should be equal to one \n.
                regex: /
<])*>\n?/gi,
                replacement: '\n'
            },

sure we will not edit on node module but we will add the changes on expensify-common library repo

@ahmdshrif yep, that's exactly what I tried. It didn't work.
Btw, I'm testing pasting this text -
https://github.com/Expensify/App/blob/403fc28984e9d96f3127257a3dc3ac6e66a75cd6/src/components/TextInputFocusable/index.js#L263-L267

@rushatgabhane ok, GitHub uses td  and tr Tags for new line not dive as vs code .
so just add them. like this will work on both .
pre: inputString => inputString.replaceAll('
', '').replaceAll('
', '
').replaceAll('
', '
').replace('

', '
').replace('

', '
')
.replace(SLACK_SPAN_NEW_LINE_TAG + SLACK_SPAN_NEW_LINE_TAG, '

').replace(SLACK_SPAN_NEW_LINE_TAG, '

'),

@ahmdshrif thanks, that fixed it for copying form github.
Copying 1 line from vscode pastes 2 extra lines.

@ahmdshrif gentle reminder, we're trying to paste it into code blocks instead of plain text.
I hope I make sense

we can fix the extra line but what do you mean by code blocks. do we need to add a feature for code block highlight or do we just fix the  new line issue with code block past?

My bad, I was mistaken that we need to enclose the copied code into code blocks like this
copied code here

we can fix the extra line

@ahmdshrif  Perfect!

@stephanieelliott I am going OOO so I am reassigning. Job is created, just need to hire and payout when the time comes!

we can fix the extra line
@ahmdshrif Perfect!

Hey @rushatgabhane, just wanted to confirm - does this mean we want to move forward with @ahmdshrif's proposal?

@rushatgabhane
this is the final proposal that fix the extralines
 pre: inputString => inputString.replaceAll('

', '
').replaceAll('
', '
').replace('

', '
').replace('

', '
')
.replace(SLACK_SPAN_NEW_LINE_TAG + SLACK_SPAN_NEW_LINE_TAG, '

').replace(SLACK_SPAN_NEW_LINE_TAG, '

'),

@iwiznia, @rushatgabhane, @stephanieelliott Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

Hey @rushatgabhane, just wanted to confirm - does this mean we want to move forward with @ahmdshrif's proposal?

No, not yet

@ahmdshrif since you're modifying the same rule in https://github.com/Expensify/App/issues/7644,
could you please post an updated proposal which combines both for better clarity?

Posting it here too for better visibility.

Referring to your proposal here - https://github.com/Expensify/App/issues/7352#issuecomment-1048130956

@ahmdshrif small issue 😅 visit this file and try copying these lines.
https://github.com/Expensify/App/blob/403fc28984e9d96f3127257a3dc3ac6e66a75cd6/src/components/TextInputFocusable/index.js#L14-L16
I'm getting 2 line breaks instead of 1.
    maxLines: PropTypes.number,
<-- line break
<-- line break
    /** The default value of the comment box */
    defaultValue: PropTypes.string,

I think listing all the cases, and how your fix handles each case would be a good idea (easy to debug then).

Friendly bump @ahmdshrif, please submit an updated proposal if you'd like to work on this!

Triggered auto assignment to @jboniface (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.

@stephanieelliott  thanks for pinging me I will fix the extra line issue soon.

@rushatgabhane  as i mention here https://github.com/Expensify/App/issues/7352#issuecomment-1052139980 this is what we need to fix both issue .
copy code here have 2 issue : -
1- when copy form vsCode .
vs code map newline with div  so that same regex we need to fix the 7352 issue will fix  this part which is .replaceAll(/<\/div>(<[^/])/g, "
$1")
2- copy from Github .
GitHub map the code as a table and every row on the table is a line. so we just need to add a new line after each row

and i write this regex to fix this .replaceAll(/(<tr.*?<\/tr>)/g,"$1
") ;
but we still get a new line at the end so we will not map the last row to a new line.
.replaceAll('

',"")
this proposal works fine with all cases you mention and I try more cases and work perfectly.
https://user-images.githubusercontent.com/21364901/155846756-a629e844-3869-45cf-9a7e-cf569aefed6b.mov

@rushatgabhane or @iwiznia do you have input on the proposal?

Extremely sorry for the delay
@ahmdshrif we just need to handle this case - https://github.com/Expensify/App/issues/7352#issuecomment-1061093432

Sorry, but isn't this issue basically the same as https://github.com/Expensify/App/issues/7352?

@rushatgabhane  fixed here https://github.com/Expensify/App/issues/7352#issuecomment-1064036764
@iwiznia  they have some similarities but different scenarios. and we need here to handle some other cases. like td tr .

Proposal
In src/components/TextInputFocusable/index.js delete these lines
        if (types.includes(TEXT_HTML)) {
            const pastedHTML = event.clipboardData.getData(TEXT_HTML);

            event.preventDefault();
            const domparser = new DOMParser();
            const embeddedImages = domparser.parseFromString(pastedHTML, TEXT_HTML).images;

            // If HTML has img tag, then fetch images from it.
            if (embeddedImages.length > 0) {
                fetch(embeddedImages[0].src)
                    .then((response) => {
                        if (!response.ok) { throw Error(response.statusText); }
                        return response.blob();
                    })
                    .then((x) => {
                        const extension = IMAGE_EXTENSIONS[x.type];
                        if (!extension) {
                            throw new Error(this.props.translate('textInputFocusable.noExtentionFoundForMimeType'));
                        }

                        return new File([x], `pasted_image.${extension}`, {});
                    })
                    .then(this.props.onPasteFile)
                    .catch(() => {
                        const errorDesc = this.props.translate('textInputFocusable.problemGettingImageYouPasted');
                        Growl.error(errorDesc);

                        /*
                        * Since we intercepted the user-triggered paste event to check for attachments,
                        * we need to manually set the value and call the `onChangeText` handler.
                        * Synthetically-triggered paste events do not affect the document's contents.
                        * See https://developer.mozilla.org/en-US/docs/Web/API/Element/paste_event for more details.
                        */
                        this.handlePastedHTML(pastedHTML);
                    });
                return;
            }

            this.handlePastedHTML(pastedHTML);
        }

increasing bounty

Latest proposal review here - https://github.com/Expensify/App/issues/7352#issuecomment-1069723246

@rushatgabhane can you clarify the status of this issue? do we have two issues solving the same problem?

@jboniface The proposals for both #7352 and this issue, modify the same Expensimark rule. So @ahmdshrif posted a combined proposal on #7352
Anyway, I've recommended that we hire @ahmdshrif to fix this issue - https://github.com/Expensify/App/issues/7352#issuecomment-1074459989

@ahmdshrif could you please repost your proposal here? And slightly customize it for this issue.
It'll make it easier for @ iwiznia to review it. Thanks a lot!

sure  @rushatgabhane
@iwiznia  we have an issue here when copying text from the text editor  because it uses div to make a new line
and that is what we try to fix on both issues.
so I add the regex before here and it work fine but did not work on the second issue so I replace it with some logic explained here https://github.com/Expensify/App/issues/7352#issuecomment-1064853399 to work with both.
but we still have here another issue here  when copy code form github . github use  and
 (tables) to map new lines  and to fix that i add this regex .replaceAll(/(<tr.*?<\/tr>)/g,"$1
")
I also explain this regex and the first regex (we replace it ) on this comment https://github.com/Expensify/App/issues/7644#issuecomment-1052144583
as samery we will fix mapping dev on the second issue but we still need to fix copy form github buy use this regex
.replaceAll(/(<tr.*?<\/tr>)/g,"$1
")
thanks .

🎀 👀 🎀 C+ reviewed
@iwiznia I know this isn't very pretty, but the only other alternative is to use a third party markdown-html parser. But as Rory and I discussed, it's a very long term project that may or may not come into fruition.
I recommend @ahmdshrif's proposal.

Where exactly will that code go? (sorry if it is linked somewhere, but could not find it following the linked comments)
Wouldn't those regexes also match other things? Like what if I am pasting an HTML string? Ie: what if what I am pasting is
Hello!
, would that get replaced or is the escaping of html characters happening before and thus would work?

https://github.com/Expensify/App/issues/7644#issuecomment-1078170122
@ahmdshrif ^^

sorry @iwiznia  for making it confusing
I mention it early here https://github.com/Expensify/App/issues/7644#issuecomment-1035605998  and forget to add it to the final proposal
when past text we check if it contains HTML format and if yes we pare it with htmlToMarkdown
https://github.com/Expensify/App/blob/403fc28984e9d96f3127257a3dc3ac6e66a75cd6/src/components/TextInputFocusable/index.js#L244
so the library has its own mechanism to just pares the HTML style tags.
but the issue in the library is it only maps br  tag to newline and not handle the table case
so we only will add this line to the HTML parser role
https://github.com/Expensify/expensify-common/blob/main/lib/ExpensiMark.js#L197
 {
                name: 'newline',

                // Replaces open and closing

 tags with a single

                // Slack uses special  tag for empty lines instead of
 tag
              pre: inputString => inputString.replace('

', '
').replace('

', '
')
+               .replaceAll(/(<tr.*?<\/tr>)/g,"$1
")
                .replace(SLACK_SPAN_NEW_LINE_TAG + SLACK_SPAN_NEW_LINE_TAG, '

').replace(SLACK_SPAN_NEW_LINE_TAG, '

'),

                // Include the immediately followed newline as `<br>\n` should be equal to one \n.
                regex: /
<])*>\n?/gi,
                replacement: '\n'
            },

for div handling l already implement it on this PR https://github.com/Expensify/expensify-common/pull/436/files
copy you example : -

copy form github

Got it, thanks for clarifying! The proposal makes sense @jboniface please hire @ahmdshrif

@ahmdshrif please apply here https://www.upwork.com/ab/applicants/1491819288094076928/applicants

@jboniface  the link does not work

@ahmdshrif oops, i sent you the link for the proposal reviews 😵💫
it's this: https://www.upwork.com/jobs/~0103d56637d4154cbd

pr ready here https://github.com/Expensify/expensify-common/pull/436

All set here, PR merged

PR #8743 hit production 2 days ago.

Bump @jboniface

@rushatgabhane please apply for the C+ payment https://www.upwork.com/jobs/~0103d56637d4154cbd

@jboniface unfortunately, that job is no longer available.

Thanks!

@ahmdshrif has been paid (by Jeremy), is there a reason he wasn't assigned to this issue?
@rushatgabhane I hired you, let me know when you've accepted and we'll pay.
https://www.upwork.com/jobs/~01b9b2c4c22ea68332

is there a reason he wasn't assigned to this issue

proly a small boo boo https://github.com/Expensify/App/issues/7644#issuecomment-1079277063
@mallenexpensify accepted, thank you!

Paid @rushatgabhane $1k for C+
Thanks for the help y'all!


AIP-128, AIP-156, AIP-164, AIP 148: Singleton resources in Declarative Friendly APIs have unexpected warnings

When using Declarative Friendly API AIP-128 and Singleton Resources AIP-156 at the same time the linter raises warnings that likely it shouldn't.

AIP 164 Declarative Friendly entities typically require soft-delete and undelete, but this will often not make sense for Singleton resources.  In particular AIP-156 requires that Singletons not define a Delete verb:

Singleton resources must not define the Create, List, or Delete standard methods. The singleton is implicitly created or deleted when its parent is created or deleted.

AIP 148 Declarative Friendly entities require delete_time (since resource with soft-delete and undelete defined are expected to have it), but Singleton are expected to not have delete verbs as above.

Ohoho, thanks for this report! We have written a lot of these rules but they have not seen very much real world use. Will get these addressed ASAP.

core::aip0164::declarative-friendly-required only reports a problem if a declarative-friendly resource has a Delete method but no Undelete method, so I don't think it needs to be changed:
https://github.com/googleapis/api-linter/blob/c18f3154970793d9b984a376344d44e73ee4eb39/rules/aip0164/declarative_friendly_required.go#L24
But I'm working on a fix for to prevent core::aip0148::declarative-friendly-fields from requiring a delete_time field for singleton resources.


[SC-879] Deploy OffchainOracle via create3

Codecov Report
Patch and project coverage have no change.

Comparison is base (e76f09a) 78.33% compared to head (151a801) 78.33%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master      #73   +/-   ##
=======================================
  Coverage   78.33%   78.33%
=======================================
  Files          26       26
  Lines         734      734
  Branches      166      166
=======================================
  Hits          575      575
  Misses        159      159

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Make release-tab non dependent of more-dropdown

LINKED ISSUES:
Resolves #3652

TEST URLS:
None

SCREENSHOT:
None

The dropdown is native, we’re not creating it. Perhaps the only problem is that there’s no native separator in there.

PR title needs update than it’s ok if it works

How is it now?


Support capture groups for smart renaming

Unless I'm being silly, I don't think it's possible to rename the first space (" ") character in a title with a hyphen ("-") for example - as any capture which anchors to the start of the string will mean that from the start of the string is replaced.
I then guessed that perhaps this plugin allowed you to use capture groups to smart replace, so my above scenario might be solved by:
Capture: ^(?<start>[a-zA-Z0-9]*)
Replace: ^\k<start>-
This also didn't work, hence I'm suggesting it here! If my understanding of regex is correct I imagine it should be fairly straightforward to implement.
Thank you!
Dan

Look at https://github.com/OlegLustenko/obsidian-bulk-rename/issues/20. You have to use a different kind of capture groups.


MZI crosstalk heating experiments

To benchmark femwell

vary gap between heated and unheated arm to measure heat diffusion length

vary waveguide width (smaller + wider) to measure efficiency w/ mode confinement

@HelgeGehring

will require https://github.com/gdsfactory/gdsfactory/pull/2238 or some similar fix @joamatab


Add "What do I do next" to SameDiff

added but we might want to consider some design improvements for the SameDiff results page as it's a little messy and overwhelming


cpu-z-arm64: add v1.01

Help needed
The package of arm64 version does not include config file, therefore a pre_install script was added to ensure the file before persist.
However, I have no WoA device to test the manifest, I'll appreciate if someone can come with help ^^

Closes #12698

[x] I have read the Contributing Guide.

/verify

Why a separate manifest? Could this be added to the existing cpu-z manifest?

Why a separate manifest? Could this be added to the existing cpu-z manifest?

As said in related issue

This version uses different version numbers and URIs from x86/amd64 version, so there needs to be a separated manifest.


[CPP] make sure the cnc.dat have valid file length before init mpsc

Without this patch, the following error arise:
74/74 Test #74: archiveTest .........................***Failed   10.65 sec
Running main() from D:\a\aeron\aeron\cppbuild\Release\thirdparty\gmock\src\gmock\googletest\src\gtest_main.cc
[==========] Running 13 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 13 tests from AeronArchiveTest
[ RUN      ] AeronArchiveTest.shouldSpinUpArchiveAndShutdown
Shutdown Archive...
ArchivingMediaDriver PID 476
Java 1.8
C:/Program Files/Java/jdk8u262-b10/bin/java.exe
D:/a/aeron/aeron/aeron-all/build/libs/aeron-all-1.29.1-SNAPSHOT.jar
D:/a/aeron/aeron/cppbuild/Release/aeron-archive/src/test/cpp/archive
Shutting down PID 476
Waiting finished: 0
Deleting C:\Users\RUNNER~1\AppData\Local\Temp\aeron-runneradmin
Deleting D:/a/aeron/aeron/cppbuild/Release/aeron-archive/src/test/cpp/archive
[       OK ] AeronArchiveTest.shouldSpinUpArchiveAndShutdown (1075 ms)
[ RUN      ] AeronArchiveTest.shouldBeAbleToConnectToArchive
Exception in thread "main" java.nio.file.FileSystemException: C:\Users\RUNNER~1\AppData\Local\Temp\aeron-runneradmin\cnc.dat: The process cannot access the file because it is being used by another process.

Signed-off-by: Yonggang Luo<EMAIL_ADDRESS>

I've taken a different approach to this is similar to what works in Java.


add catch for TimeoutException

Add catch for TimeoutException and enable timeout configuration via environmnet variable. Use retry decorator in non-invasive functions.
related: #103

Maybe I missed some discussions on this, but I don't quite understand the reason of having CommandExecutionFailed and ActionFailedToRun. What is the behavior exactly when running actions or commands in retry hook?

It's only to be able to catch it and not retry any action or command, only retry if waiting for action failed or getting action results failed, etc. We do not want to trigger single action multiple times, it can be dangerous.


Should nocache also mean no-precompile cache?

As implemented, nocache option disables use of precompiled cache as well. Is that expected behavior?
The reason to bring this up - With nocache option enabled some of the test cases (like MiniAmiq) fail. I narrowed it down to order in which the files are preprocessed. Also, MiniAmiq does have any input files except for the UVM files. Is that correct?

No it should not. Precompiled packages require very seldom to be compiled (New build, change in grammar).
When changing the grammar, I don't want to re-incur the cost of pre-compiled packages unless necessary (After finalizing the changes).

MiniAmiq contains the UVM package plus the sva files (not part of default UVM).


Link external articles into blog page

The blog page should also display articles that I have written externally, with the link opening to the external source if clicked on

Closing as completed


JsonExtensionData emits duplicate properties during serialization if named property also exists

JsonExtensionData appears to emit duplicate elements during serialization if a key in the JsonExtensionData dictionary also exists as a named property.
I would have expected (by default at least) for the named member to be serialized and the extension data property to be ignored.
Simple repro:
using System;
using System.Collections.Generic;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace Test
{
    public class Program
    {
        [JsonObject]
        class SomeContract
        {
            [JsonProperty(PropertyName = "foo")]
            public string Foo { get; set; }

            [JsonExtensionData]
            public IDictionary<string, JToken> AdditionalData { get; set; }
        }

        public static void Main(string[] args)
        {
            Console.WriteLine(JsonConvert.SerializeObject(new SomeContract
            {
                Foo = "good",
                AdditionalData = new Dictionary<string, JToken> { { "foo", "bad" } }
            }));
        }
    }
}

Output:
{"foo":"good","foo":"bad"}
A more realistic example that we encountered is:

Deserialize to type with JsonExtensionData
Construct other type with JsonExtensionData (possibly a derived type), copying extension data
Serialize other type

I'm ok with it working like this. To change this behavior the serializer would need to keep track of each property being written and I don't think the overhead is worth it.

Noted, thanks. At least the behavior is now documented here, although updating the official docs probably wouldn't hurt either.

Done https://github.com/JamesNK/Newtonsoft.Json/commit/59618df631c10be4d25cfa7224bb18bdbde72a9b


feat: Adding some extra config to config.sh script

Adding some additional exposure to config.sh script to expose on .envrc configurations that may be interesting to consider when deploying a new testnet.
Also included a small fix on Makefile caused by a dead npm script: https://github.com/ethereum-optimism/optimism/pull/10821

Guide I used for deploying from this branch:
https://gist.github.com/jcortejoso/55baa06eb309199b84219f98d4817a67


Cloud DNS migration for KCC GKE cluster in sept 2023

during https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/296#issuecomment-1663196129
Starting from September 2023, all GKE Autopilot clusters (minimum version to be determined in August 2023) will migrate to Cloud DNS as the default DNS provider (in replacement of kube-dns), at no extra charge.
However, if you plan to deploy or migrate GKE Autopilot clusters to an Assured Workloads  folder with an Impact Level 4 (IL4)  compliance regime, please reach out to your account manager to get more details about your migration path.
Learn more about Cloud DNS for GKE
https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-dns?_ga=2.151718137.-1143133292.1680630953

I don't believe this should have any impact on the LZ Config Controller as we're not deploying any workloads or services. Are there any particular areas where you have concerns about how this might impact things?