cluster remote search

Hello,
remote cluster is up and available but I can't see indices.
About doc https://opendistro.github.io/for-elasticsearch-docs/docs/security/cross-cluster-search/
where have to put this lines ?
Is it possible to give me a full example ?
Regards

Hi @hihouhou, those lines are from roles.yml, but as far as I know, you should be able to create a role in Kibana (using an admin user) that has those same permissions. Can you try something like the following?

I tried this but nothing :-(
proxy_elas ~ 0 # curl -XGET -H 'Content-Type: application/json' https://<IP_ADDRESS>:9200/_remote/info?pretty -u admin:admin
{
  "toto" : {
    "seeds" : [
      "toto.service.consul:9300"
    ],
    "http_addresses" : [
      "XX.XX.XX.XX:9200",
      "XX.XX.XX.XX:9200",
      "XX.XX.XX.XX:9200"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

remote_elas ~ 130 # curl -sX GET "https://<IP_ADDRESS>:9200/_cluster/health" -u admin:admin | jq .
{
  "cluster_name": "remote",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 76,
  "active_shards": 153,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

remote_elas ~ 0 # curl -X GET -s "https://<IP_ADDRESS>:9200/_opendistro/_security/api/roles/humanresources" -u admin:admin | jq .
{
  "humanresources": {
    "cluster": [
      "CLUSTER_COMPOSITE_OPS_RO"
    ],
    "indices": {
      "hihouhou": {
        "*": [
          "READ",
          "indices:admin/shards/search_shards ¸ needed for CCS"
        ]
      }
    }
  }
}

remote_elas ~ 130 # curl -X GET "https://<IP_ADDRESS>:9200/_cat/indices" -u admin:admin
...
green open hihouhou-2019.05.19          05PjMQMMS0uV4gSkmHBOHA 5 1     0 0   2.5kb    1.2kb
green open hihouhou-2019.05.18          tH6zCimOSXeDFW05VNTv2w 5 1     0 0   2.5kb    1.2kb
...

proxy_elas ~ 0 # curl -ks 'https://<IP_ADDRESS>:9200/_cluster/health?pretty=true' -u admin:admin
{
  "cluster_name" : "proxy",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 138,
  "active_shards" : 277,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Hello,
You can try the following - start 2 elasticsearch nodes, one on 9250 and 9000 .. You may use my docker compose file:
version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  kibana-1:
    image: amazon/opendistro-for-elasticsearch-kibana:0.9.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      ELASTICSEARCH_HOSTS: https://odfe-node1:9200
    networks:
      - odfe-net

  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster-2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
    ports:
      - 9250:9200
      - 9700:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  kibana-2:
    image: amazon/opendistro-for-elasticsearch-kibana:0.9.0
    container_name: odfe-kibana-2
    ports:
      - 5603:5601
    expose:
      - "5603"
    environment:
      ELASTICSEARCH_URL: https://odfe-node2:9200
      ELASTICSEARCH_HOSTS: https://odfe-node2:9200
    networks:
      - odfe-net

volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net:

After running docker-compose -up, you need to find the IP address of each container, do the following:
➜ ~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cbd47a0625e0 amazon/opendistro-for-elasticsearch-kibana:0.9.0 "/usr/local/bin/kiba…" About a minute ago Up 57 seconds 5603/tcp, <IP_ADDRESS>:5603→5601/tcp odfe-kibana-2
381bef7218fb amazon/opendistro-for-elasticsearch:0.9.0 "/usr/local/bin/dock…" About a minute ago Up 57 seconds <IP_ADDRESS>:9200→9200/tcp, <IP_ADDRESS>:9600→9600/tcp, 9300/tcp odfe-node1
c5779c179ca7 amazon/opendistro-for-elasticsearch-kibana:0.9.0 "/usr/local/bin/kiba…" About a minute ago Up 58 seconds <IP_ADDRESS>:5601→5601/tcp odfe-kibana
da6f5e51edfa amazon/opendistro-for-elasticsearch:0.9.0 "/usr/local/bin/dock…" About a minute ago Up 58 seconds 9300/tcp, <IP_ADDRESS>:9250→9200/tcp, <IP_ADDRESS>:9700→9600/tcp odfe-node2

➜  ~ docker inspect -f 'range .NetworkSettings.Networks.IPAddressend' 381bef7218fb
<IP_ADDRESS>
➜  ~ docker inspect -f 'range .NetworkSettings.Networks.IPAddressend' da6f5e51edfa
<IP_ADDRESS>

Notice my 9200 ES container's ip address is <IP_ADDRESS>, and the 9250 is <IP_ADDRESS>
In this example, we will try to search the 9200 container through 9250, so add the following to 9250 ES container.
➜  ~ curl -k -XPUT -H 'Content-Type: application/json' https://admin:admin@localhost:9250/_cluster/settings -d '
{
    "persistent": {
        "search.remote": {
            "odfe-cluster": {
                "seeds": ["<IP_ADDRESS>:9300"]
            }
        }
    }
}'
{"acknowledged":true,"persistent":{"cluster":{"remote":{"odfe-cluster":{"seeds":["<IP_ADDRESS>:9300"]}}}},"transient":{}}%

After this step, you should be able to
➜ ~ curl -XGET -k https://admin:admin@localhost:9250/_remote/info\?pretty (https://admin:admin@localhost:9250/_remote/info/?pretty)
{
"odfe-cluster" : {
"seeds" : [
"<IP_ADDRESS>:9300"
],
"http_addresses" : [
"<IP_ADDRESS>:9200"
],
"connected" : true,
"num_nodes_connected" : 1,
"max_connections_per_cluster" : 3,
"initial_connect_timeout" : "30s",
"skip_unavailable" : false
}
}

Let's now create a searchable index on 9200, and add some data(optional)
➜ ~ curl -XPUT -k https://admin:admin@localhost:9200/books
{"acknowledged":true,"shards_acknowledged":true,"index":"books"}%

NOW, let's do a cross cluster search using admin credentials, it will be able to make the call across clusters - the reason this works is because I am passing in admin credentials.
➜ ~ curl -XGET -k https://admin:admin@localhost:9250/odfe-cluster:books/_search
{"took":207,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"_clusters":{"total":1,"successful":1,"skipped":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"odfe-cluster:books","_type":"fiction","_id":"1","_score":1.0,"_source":{"name": ...

What if you didn't want to use admin credentials - in that case you need do the following:
First, let's create a user, call it booksuser on 9250 and 9200
curl -XPUT -k https://admin:admin@localhost:9250/_opendistro/_security/api/internalusers/booksuser  -H 'Content-Type: application/json' -d '{"password":"password"}'
{"status":"CREATED","message":"'booksuser' created."}%

curl -XPUT -k https://admin:admin@localhost:9200/_opendistro/_security/api/internalusers/booksuser  -H 'Content-Type: application/json' -d '{"password":"password"}'
{"status":"CREATED","message":"'booksuser' created."}%

Now, let's run the same command we used to cross cluster search using this new booksuser.
curl -XGET -k https://booksuser:password@localhost:9250/odfe-cluster:books/_search
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"},"status":403}%

You can see what error message is telling you .. you need a role providing those index permissions to the booksuser, you may do so like this - NOTE - you need to create both role and the role mapping.
➜  ~ curl -XPUT -k https://admin:admin@localhost:9250/_opendistro/_security/api/roles/booksrole -H 'Content-Type: application/json' -d '{"indices" : {"books" : {"" : [ "indices:admin/shards/search_shards","indices:data/read/search"]}}}' -i -v   Trying ::1...

* TCP_NODELAY set
* Connected to localhost (::1) port 9250 (#0)
* WARNING: disabling hostname validation also disables SNI.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: node-0.example.com (http://node-0.example.com/)
* Server auth using Basic with user 'admin'

> PUT /_opendistro/_security/api/roles/booksrole HTTP/1.1
> Host: localhost:9250
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.54.0
> Accept: /
> Content-Type: application/json
> Content-Length: 100
>

* upload completely sent off: 100 out of 100 bytes

< HTTP/1.1 201 Created
HTTP/1.1 201 Created

*➜  ~ curl -XPUT -k https://admin:admin@localhost:9250/_opendistro/_security/api/rolesmapping/booksrole -H 'Content-Type: application/json' -d '{"users" : ["booksuser"]}' -i -v*

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9250 (#0)
* WARNING: disabling hostname validation also disables SNI.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: node-0.example.com (http://node-0.example.com/)
* Server auth using Basic with user 'admin'

> PUT /_opendistro/_security/api/rolesmapping/booksrole HTTP/1.1
> Host: localhost:9250
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.54.0
> Accept: /
> Content-Type: application/json
> Content-Length: 25
>

* upload completely sent off: 25 out of 25 bytes

< HTTP/1.1 201 Created
HTTP/1.1 201 Created

Now, you will be able to run the command and do cross cluster search ->
➜  ~ curl -XGET -k https://booksuser:password@localhost:9250/odfe-cluster:books/_search
{"took":63,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"_clusters":{"total":1,"successful":1,"skipped":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"odfe-cluster:books","_type":"fiction","_id":"1","_score":1.0,"_source":{"name": "H....

nothing found :-(
hihouhou@test-opendistro:~$ docker ps
CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
b6bfa6b71cf8        amazon/opendistro-for-elasticsearch:0.9.0          "/usr/local/bin/dock…"   2 minutes ago       Up About a minute   9300/tcp, <IP_ADDRESS>:9250->9200/tcp, <IP_ADDRESS>:9700->9600/tcp   odfe-node2
944884862d4c        amazon/opendistro-for-elasticsearch:0.9.0          "/usr/local/bin/dock…"   3 minutes ago       Up About a minute   <IP_ADDRESS>:9200->9200/tcp, <IP_ADDRESS>:9600->9600/tcp, 9300/tcp   odfe-node1
88918eedc4ae        amazon/opendistro-for-elasticsearch-kibana:0.9.0   "/usr/local/bin/kiba…"   About an hour ago   Up About a minute   <IP_ADDRESS>:5601->5601/tcp                                     odfe-kibana
519017727e69        amazon/opendistro-for-elasticsearch-kibana:0.9.0   "/usr/local/bin/kiba…"   About an hour ago   Up About a minute   5603/tcp, <IP_ADDRESS>:5603->5601/tcp                           odfe-kibana-2

hihouhou@test-opendistro:~$ docker inspect b6bfa6b71cf8 | grep '                    "IPAddress'
                    "IPAddress": "<IP_ADDRESS>",
hihouhou@test-opendistro:~$ docker inspect 944884862d4c | grep '                    "IPAddress'
                    "IPAddress": "<IP_ADDRESS>",

hihouhou@test-opendistro:~$ curl -k -XPUT -H 'Content-Type: application/json' https://admin:admin@localhost:9250/_cluster/settings -d '
> {
>     "persistent": {
>         "search.remote": {
>             "odfe-cluster": {
>                 "seeds": ["<IP_ADDRESS>:9300"]
>             }
>         }
>     }
> }'
{"acknowledged":true,"persistent":{"cluster":{"remote":{"odfe-cluster":{"seeds":["<IP_ADDRESS>:9300"]}}}},"transient":{}}hihouhou@test-opendistro:~$ curl -XGET -k https://admin:admin@localhost:9250/_remote/info\?pretty
{
  "odfe-cluster" : {
    "seeds" : [
      "<IP_ADDRESS>:9300"
    ],
    "http_addresses" : [
      "<IP_ADDRESS>:9200"
    ],
    "connected" : true,
    "num_nodes_connected" : 1,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

hihouhou@test-opendistro:~$ curl -XPUT -k https://admin:admin@localhost:9200/books
{"acknowledged":true,"shards_acknowledged":true,"index":"books"}hihouhou@test-opendistro:~$

hihouhou@test-opendistro:~$ curl -k https://admin:admin@localhost:9200/_cat/indices
yellow open books                        ny145AOpTgK2iHNHQyt90Q 5 1 0 0  1.1kb  1.1kb
yellow open security-auditlog-2019.05.23 1U6SgWpIQWKL7K-Tq_2YcA 5 1 2 0 24.4kb 24.4kb
green  open .kibana_1                    q7UneMIBTkOK3mFe9t-8yQ 1 0 0 0   261b   261b
green  open .opendistro_security         ySCEelghTRSVQlH7UJJ-IA 1 0 5 0 31.8kb 31.8kb

hihouhou@test-opendistro:~$ curl -XGET -sk https://admin:admin@localhost:9250/odfe-cluster:books/_search | jq .
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "index_uuid": "_na_",
        "resource.type": "index_or_alias",
        "resource.id": "books",
        "index": "books"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "index_uuid": "_na_",
    "resource.type": "index_or_alias",
    "resource.id": "books",
    "index": "books"
  },
  "status": 404
}

I tried with specific user : same result
hihouhou@test-opendistro:~$ curl -XPUT -k https://admin:admin@localhost:9250/_opendistro/_security/api/internalusers/booksuser  -H 'Content-Type: application/json' -d '{"password":"password"}'
{"status":"CREATED","message":"'booksuser' created."}hihouhou@test-opendistro:~$
hihouhou@test-opendistro:~$ curl -XPUT -k https://admin:admin@localhost:9200/_opendistro/_security/api/internalusers/booksuser  -H 'Content-Type: application/json' -d '{"password":"password"}'
{"status":"CREATED","message":"'booksuser' created."}hihouhou@test-opendistro:~$
hihouhou@test-opendistro:~$ curl -XGET -k https://booksuser:password@localhost:9250/odfe-cluster:books/_search
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"},"status":403}hihouhou@test-opendistro:~$

hihouhou@test-opendistro:~$ curl -XPUT -k https://admin:admin@localhost:9200/_opendistro/_security/api/roles/booksrole -H 'Content-Type: application/json' -d '{"indices" : {"books" : {"" : [ "indices:admin/shards/search_shards","indices:data/read/search"]}}}' -i -v
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
*  start date: Apr 22 03:43:47 2018 GMT
*  expire date: Apr 19 03:43:47 2028 GMT
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> PUT /_opendistro/_security/api/roles/booksrole HTTP/1.1
> Host: localhost:9200
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 99
>
* upload completely sent off: 99 out of 99 bytes
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-6.7.1-2f32220 "Usage of internal versioning for optimistic concurrency control is deprecated and will be removed. Please use the `if_seq_no` and `if_primary_term` parameters instead. (request for index [.opendistro_security], type [security], id [roles])"
Warning: 299 Elasticsearch-6.7.1-2f32220 "Usage of internal versioning for optimistic concurrency control is deprecated and will be removed. Please use the `if_seq_no` and `if_primary_term` parameters instead. (request for index [.opendistro_security], type [security], id [roles])"
< content-type: application/json; charset=UTF-8
content-type: application/json; charset=UTF-8
< content-length: 48
content-length: 48

<
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
{"status":"OK","message":"'booksrole' updated."}hihouhou@test-opendistro:~$
hihouhou@test-opendistro:~$
hihouhou@test-opendistro:~$
hihouhou@test-opendistro:~$
hihouhou@test-opendistro:~$ curl -XPUT -k https://admin:admin@localhost:9250/_opendistro/_security/api/roles/booksrole -H 'Content-Type: application/json' -d '{"indices" : {"books" : {"" : [ "indices:admin/shards/search_shards","indices:data/read/search"]}}}' -i -v
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9250 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
*  start date: Apr 22 03:43:47 2018 GMT
*  expire date: Apr 19 03:43:47 2028 GMT
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> PUT /_opendistro/_security/api/roles/booksrole HTTP/1.1
> Host: localhost:9250
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 99
>
* upload completely sent off: 99 out of 99 bytes
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-6.7.1-2f32220 "Usage of internal versioning for optimistic concurrency control is deprecated and will be removed. Please use the `if_seq_no` and `if_primary_term` parameters instead. (request for index [.opendistro_security], type [security], id [roles])"
Warning: 299 Elasticsearch-6.7.1-2f32220 "Usage of internal versioning for optimistic concurrency control is deprecated and will be removed. Please use the `if_seq_no` and `if_primary_term` parameters instead. (request for index [.opendistro_security], type [security], id [roles])"
< content-type: application/json; charset=UTF-8
content-type: application/json; charset=UTF-8
< content-length: 48
content-length: 48

<
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
{"status":"OK","message":"'booksrole' updated."}

hihouhou@test-opendistro:~$ curl -sXGET -k https://booksuser:password@localhost:9250/odfe-cluster:books/_search | jq .
{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"
      }
    ],
    "type": "security_exception",
    "reason": "no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"
  },
  "status": 403
}

Hi @hihouhou, a couple things:

In your latest reply, it looks like you added the role on both clusters, but the mapping on neither. You need the role and mapping to be present only on 9200. The user needs to be present on both clusters. It might be easier to use Kibana to verify that the role and mapping exist where you expect them to exist.

Can you verify consistency between each cluster, IP, and node name to make sure everything is being routed correctly? I can't quite tell if you added the correct seed or not or if your requests are going to the coordinating cluster to the remote cluster, if you're indexing to the right cluster, etc.

Hi @aetter, thanks for sharing these informations, they are really helpful.

Hi @ptran32, authentication takes place on the coordinating node, but final authorization to view the data takes place on the remote. That's why, in this case, both nodes need the user account, but only one node (the remote) needs the security role/mapping.
To get all the features of ODFE, like role-based access control and document- and field-level security, you'll need the Security plugin on all nodes. If you're looking at access control as all-or-nothing (i.e. if the user successfully authenticated, he/she has full privileges), then your configuration might work, but I haven't tested it and doubt anyone else has. 😄
In general, the forums are a great place for broader discussions or sharing of best practices, and GitHub is a good place for specific, reproducible problems and feature requests.

Thanks for you clear reply @aetter !

Hi @aetter, cluster remote search works well with a new created user, but my biggest concern is trying to be able to create an index pattern with a remote index from Kibana.
I started from the beginning and trying to make it works as admin user.
It looks like Kibana can't handle remote search or am I missing something in the configuration.
Opendistro Version : 6.7.1
What I expect : Create an index pattern from Kibana which target the index located on remote node
Current behaviour : Cannot add index pattern for remote index (no error message in kibana or remote node)
User logged in Kibana : admin:admin . I expected from this user to be able to perform remote_search on all indexes in the cluster
Here are the steps to reproduce the issue:
1) Create a docker-compose.yml with this content and start the stack with "docker-compose up":
version: '3'
services:
odfe-node1:
image: amazon/opendistro-for-elasticsearch:0.9.0
container_name: odfe-node1
environment:
- cluster.name=odfe-cluster
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- opendistro_security.ssl.http.enabled=false
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- odfe-data1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- odfe-net
odfe-ccs:
image: amazon/opendistro-for-elasticsearch:0.9.0
container_name: odfe-ccs
environment:
- cluster.name=odfe-ccs-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- opendistro_security.ssl.http.enabled=false
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- odfe-data3:/usr/share/elasticsearch/data
networks:
- odfe-net
kibana:
image: amazon/opendistro-for-elasticsearch-kibana:0.9.0
container_name: odfe-kibana
ports:
- 5601:5601
expose:
- "5601"
environment:
ELASTICSEARCH_URL: http://odfe-node1:9200
ELASTICSEARCH_HOSTS: https://odfe-node1:9200
networks:
- odfe-net
volumes:
odfe-data1:
odfe-data2:
odfe-data3:
networks:
odfe-net:
2) From another terminal, create an index with document on remote node :
docker exec -it odfe-ccs sh
curl -u admin:admin -k http://localhost:9200/my_index/my_type -curl -H 'Content-Type: application/json' -d '{"user":"Phil","message":"Hello World!"}'
3) Verify the index has been created :
curl -u admin:admin http://localhost:9200/my_index
4) On kibana (localhost:5601), add the remote node
PUT _cluster/settings
{
"persistent": {
"cluster": {
"remote": {
"cluster_ccs": {
"seeds": [
"odfe-ccs:9300"
]
}
}
}
}
}
5) Verifiy the remote node has been added and connected
GET _remote/info
The output should be like below.
{
"cluster_ccs" : {
"seeds" : [
"odfe-ccs:9300"
],
"http_addresses" : [
"<IP_ADDRESS>:9200"
],
"connected" : true,
"num_nodes_connected" : 1,
"max_connections_per_cluster" : 3,
"initial_connect_timeout" : "30s",
"skip_unavailable" : false
}
}
6) Do a remote search from Kibana to reach the remote index
GET cluster_ccs:my_index/_search
The output should be like below.
{
"took" : 569,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"_clusters" : {
"total" : 1,
"successful" : 1,
"skipped" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [
{
"_index" : "cluster_ccs:my_index",
"_type" : "my_type",
"_id" : "tWwoK2sBjdz1gCR7UH5d",
"_score" : 1.0,
"_source" : {
"user" : "Phil",
"message" : "Hello World!"
}
},
{
"_index" : "cluster_ccs:my_index",
"_type" : "my_type",
"_id" : "cBUaK2sBdezo3LXVgNQN",
"_score" : 1.0,
"_source" : {
"user" : "Phil",
"message" : "Hello World!"
}
}
]
}
}
7) Try to create an index pattern for the remote index on kibana with these syntaxes (I tried each one of these :
cluster_ccs:my_index*
cluster_ccs:my_index
cluster_ccs:*
My index never appears on Kibana.
This is a behaviour that I don't have on a classic elasticsearch and kibana.
Thank you by advance for you help.

I am also experiencing the same behaviour in Kibana. CCS works from cli/dev tools, but not when creating an index pattern.
I noticed in /var/log/messages, when you try and add the index pattern, it’s doing a post to:
/elasticsearch/remote_cluster:*/_search?ignore_unavailable=true 200
Is the fact that it’s prepending /elasticsearch causing the issue?

I have the same issue, CCS is working fine except when creating an index-pattern in Kibana.
I think I have narrowed what request is not working.
Request done by Kibana to find the indices, it's an aggregation on the field _index :
POST /remote_cluster:*/_search?ignore_unavailable=true
{
    "size":0,
    "aggs":{
        "indices":{
            "terms":{
                "field":"tag.audience",
                "size":200
            }
        }
    }
}

Somehow , when doing an aggregation on the field _index, it gets masked (empty buckets in response) :
{
    "took":93,
    "timed_out":false,
    "_shards":{
        "total":178,
        "successful":178,
        "skipped":0,
        "failed":0
    },
    "_clusters":{
        "total":1,
        "successful":1,
        "skipped":0
    },
    "hits":{
        "total":0,
        "max_score":0.0,
        "hits":[]
    },
    "aggregations":{
        "indices":{
            "doc_count_error_upper_bound":0,
            "sum_other_doc_count":0,
            "buckets":[]
        }
    }
}

But when I modify the request and add another aggregations, both aggregation pass through and the indices are well returned.
Request with aggretions on _index plus another field :
POST /remote_cluster:*/_search?ignore_unavailable=true
{
    "size":0,
    "aggs":{
        "indices":{
            "terms":{
                "field":"tag.audience",
                "size":200
            }
        },
        "audience":{
            "terms":{
                "field":"tag.audience",
                "size":200
            }
        }
    }
}

Response :
{
    "took":129,
    "timed_out":false,
    "_shards":{
        "total":178,
        "successful":178,
        "skipped":0,
        "failed":0
    },
    "_clusters":{
        "total":1,
        "successful":1,
        "skipped":0
    },
    "hits":{
        "total":7022547,
        "max_score":0.0,
        "hits":[
        ]
    },
    "aggregations":{
        "audience":{
            "doc_count_error_upper_bound":0,
            "sum_other_doc_count":0,
            "buckets":[
                {
                    "key":"team1",
                    "doc_count":4437001
                }
            ]
        },
        "indices":{
            "doc_count_error_upper_bound":0,
            "sum_other_doc_count":0,
            "buckets":[
                {
                    "key":"remote_cluster:logs_2019.06.07",
                    "doc_count":4437001
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.06.06",
                    "doc_count":383333
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.06.05",
                    "doc_count":365588
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.06.04",
                    "doc_count":348252
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.06.03",
                    "doc_count":330876
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.06.02",
                    "doc_count":313500
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.06.01",
                    "doc_count":296119
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.05.31",
                    "doc_count":279144
                },
                {
                    "key":"remote_cluster:.monitoring-es-6-2019.06.07",
                    "doc_count":268734
                }
            ]
        }
    }
}

So if i'm right there seems to be a bug with CCS and aggregations on the "_index" field.

I'm no expert but it seems the problems comes from this file :
github.com/opendistro-for-elasticsearch/security/src/main/java/com/amazon/opendistroforelasticsearch/security/privileges/TermsAggregationEvaluator.java
In the function evaluate() there is a bunch of code that execute for exactly the request that Kibana is doing !
It modifies the request to add filters on the indices where the user has permissions.
The problem with CCS is that this code is adding filters based on indices present on the local cluster, so it doesn't match the indices on the remote cluster, hence the empty response.

Good news.
Cross cluster works now with :

opendistro-for-elasticsearch:0.10.0
amazon/opendistro-for-elasticsearch-kibana:0.10.0

Good news.
Cross cluster works now with :

opendistro-for-elasticsearch:0.10.0
amazon/opendistro-for-elasticsearch-kibana:0.10.0

Is that also true for the Kibana Alerting plugin?

@tomerilp Unfortunately, I haven't had the chance to try the alerting plugin, and my cluster has been tear down.

Hi @hihouhou !
This has been fixed in a later release of OpenSearch. Give it a try on latest. If you still think you're having an issue and need some help, the OpenSearch Forums would be a great place to get community support.
Marking as closed, but feel free to re-open as a bug if you believe I've done this in error. :-)

thank you !!


Option output.clean ignored and violating options schema

Bug report

What is the current behavior?

output.clean option is ignored.
property output.clean is not allowed according to PHPStorm. "Validates options against webpack options schema". In reality if I build the bundles the option is just ignored without any error, so there's something weird. If I enter a wrong option it would not build.

If the current behavior is a bug, please provide the steps to reproduce.
Just try to add this to webpack.config.js:
    output: {
        clean:  true
    },

Create a dummy file touch dist/dummy and build. The dummy file remains and the JS files remain with the same date.
Same if I add a enabled subkey.

What is the expected behavior?

The output directory should be cleaned.
There should not be any Schema violation detected in PHPStorm.

Other relevant information:
webpack version:  5.30.0
Node.js version: 14.10.1
Operating System: Ubuntu under WSL
Additional tools: PHPStorm 2020.3

@chenxsan I think we need improve it and write that we only delete the files generated by the webpack, developer can use copy-webpack-plugin if they want to use external files and they will be removed

But this should still clean the output directory (default: dist) when building with webpack?

I don't understand what you mean with copy-webpack-plugin. I just want to clean the dist directory as done with CleanWebpackPlugin with minimal options. I thought clean: true would do the job, it doesn't.
Not working as described here:
https://webpack.js.org/guides/output-management/#cleaning-up-the-dist-folder

No, files from not webpack will be in place, you can store the necessary files there

Not working as described here:
https://webpack.js.org/guides/output-management/#cleaning-up-the-dist-folder

What is not working?

Not following what you mean with "files from not webpack". From the description of the option:

As you might have noticed over the past guides and code example, our /dist folder has become quite cluttered. Webpack will generate the files and put them in the /dist folder for you, but it doesn't keep track of which files are actually in use by your project.
In general it's good practice to clean the /dist folder before each build, so that only used files will be generated. Let's take care of that with output.clean option.

It seems clean does not clean anything at all. What other files than the .js files in dist is it supposed to clean?

It seems clean does not clean anything at all

Please stop say loud words :facepalm:

What other files than the generated .js files in dist is it supposed to clean?

Any from webpack

What is the problem? Please describe steps

Create a dummy file touch dist/dummy and build. The dummy file remains and the JS files remain with the same date.

Can contain any sensitive information and removing this file is not safe. You should avoid copy files without copy-webpack-plugin, using copy-webpack-plugin allows to compress/minify/changed/etc actions, it has more advantages and clean will work in this case

I'm just building very basic stuff with npm, it produces some .js files in dist. Those files are generated with webpack via npm.
output.clean does not clean the dist folder.
Why would i have to deal with copy-webpack-plugin? It's not relevant at all except for the dummy example maybe.

I'm just building very basic stuff with npm, it produces some .js files in dist. Those files are generated with webpack via npm.
output.clean does not clean the dist folder.

Please provide steps, I tested it locally right now and it works

OK thank you for nothing. I'm losing patience. I switch back to CleanWebpackPlugin.

@herrvigg What is the problem provide steps? I don't know your environment so maybe problem in other places. CleanWebpackPlugin will be deprecated and most likely may stop working in the future.

First run:

Second run:

Previously js file was removed as you can see

This is the repo: https://github.com/qtranslate/qtranslate-xt/tree/feature/webpack
In fact i was trying to get rid of CleanWebpackPlugin, by using output.clean: true.
What i do is simply "npm run build" to generate the .js files in dist but the output.clean leaves all the files as they were, while CleanWebpackPlugin would delete them before build so they are generated again.
My environment is WSL Ubuntu. Files are generated with 644 rights so that should work I guess.

In my case the target files in dist keep the same name among different runs.

output.clean keeps previously webpack assets, why you need remove them, they will be rewritten on the next write, what is use case to remove them?

We avoid unnecessary fs calls, no need to delete them and write again, just rewrite them

Oh...  f this is really the expected behavior perhaps it's the description that is quite ambiguous.

In general it's good practice to clean the /dist folder before each build,

By "cleaning a folder", i think most of developers would expect it to be emptied.
Maybe that's overkill but I want to be sure the generated files are fresh. If I use output.clean I expect the previous files to be deleted and generated again.

So output.clean is actually a bit too smart ;)

Maybe that's overkill but I want to be sure the generated files are fresh. If I use output.clean I expect the previous files to be deleted and generated again.

You don't need to worry about this, they are always fresh :smile:

By "cleaning a folder", i think most of developers would expect it to be emptied.

so I moved the problem here, we need improve docs

All right, but just a curiosity, if you change your webpack configuration (e.g. entries), how would output.clean be aware of what is generated by webpack or not? How does it differentiate for example the dummy file?

In fact it works as you described after understanding the expected behavior.
If I comment an entry, that output target is deleted with output.clean.
So I guess it's based on file information from the previous builds or with some kind of filter (e.g. .js files).

For the conclusion, I removed CleanWebpackPlugin, nice :)
An updated description in the documentation may certainly help other developers to understand this behavior.
Thank you for your help and your time!

Unfortunately I'm not able to reproduce the issue on macOS. Here's the repository I've set up https://github.com/chenxsan/webpack-js-org-issue-4806.

Clone it
Run yarn to install
Run npx webpack to build

As you can see, the dist/dummy file will be removed.
Am I misunderstanding the issue?

Thanks for testing!
Strangely I can't reproduce this with your project, in that case it removes also the dummy file indeed. How strange!
I was fine with the given explanation but now I see a difference in the behaviors of these two projects. I'll try to investigate to see if I we can find a better explanation.


Remove temp folder after installing a plugin

Since we switched from renaming to copying the plugin, we should remove the temp plugin as a cleanup procedure.
This became an issue on the DC/OS integration tests suite where the new CLI caused "No space left on device" errors.
https://jira.mesosphere.com/browse/DCOS-41483

@mesosphere-ci run integration tests


Revert "Exponential low-pass filter"

Reverts JohnEarnest/Octo#146
Would you take look at this test first?
https://johnearnest.github.io/Octo/index.html?key=DJRtrTsh
And compare it with
https://kouzeru.github.io/Octo/index.html?key=DJRtrTsh

I guess I had to elaborate my own approach of doing the filter there, as I actually done it with some analizations to get the better quality at filtering the aliasing, especially when the environment ran at 44100hz.

Ah, I get where the oscillations come from. It's a different kind of artifact that I hadn't considered. It comes from discretizing time in the pulse generator. Essentially there's a resonance between the playback frequency and the sampling frequency. It's easy to combine your idea of using a finer time control with the better low-pass filter.
Can you check this? It seems to work for me: https://codeflo.github.io/Octo/index.html?key=DJRtrTsh

Here's the oscilloscope view of the both algorithms:
https://user-images.githubusercontent.com/46141631/134085791-4e5e98cb-1592-494a-a990-04657a01955a.mp4
https://user-images.githubusercontent.com/46141631/134085802-bc1605a6-3c40-4898-97aa-648e85986ec2.mp4
By looking at the same, it's just same as my earlier implementation of the lowpass filter though!
I'm just unsure what it was called, and my last one is actually my forward step in order to give much more audio quality.
I wanted to make a playground of how the filter I had implemented there, it might take two or three days from now.
And I had a little time to explain everything, but (a subtle flex) I'm good at math.

Nice graphics.  It's simple, but should work well enough here. There's actually a lot of theory behind it.
So, is your approach based on an existing filter design (if so, any reference would probably be enough for me to look it up), or something you've come up with yourself?

I came it up with myself, but I'm sure it's existing already, it's just implemented in much simpler code.
Maybe looking at the characteristic of using acceleration and the attenuation of the velocity, I could refer it as
https://people.eecs.berkeley.edu/~boser/courses/49_sp_2019/N_filter.html

Alright, here's the playground https://kouzeru.github.io/simple-second-order-lowpass-playground/
Which validates that my approach there was based on "second-order lowpass filter", the same thing what you did. Just set the "dec" parameter to 1, and the result would be identical with the current implementation of yours, that is "first-order lowpass filter", exponential filter.

Do you have some insights after playing the playground?
If you understood how elegant fast inverse square root algorithm is, it might be proportional to this one.
The code is allowed to be used on anywhere, even to your new PR, I don't mind.
Neither you don't apply it, I also don't mind.
Now. I'm closing this.


docs: add lerna to showcase

Pre-flight checklist

[x] I have read the Contributing Guidelines on pull requests.

Motivation
It adds the Lerna website (https://lerna.js.org) to the showcase section of Docusaurus' website.
Deploy preview: https://deploy-preview-_____--docusaurus-2.netlify.app/

Ah, wait—when you force-push you forgot to pull my changes. Now I have to do it again.


NGINX example of stripping count headers

Fill in the missing example in https://postgrest.com/en/v0.4/admin.html#count-header-dos

We can probably make this how-to a bit more generic and also show how to add headers and read them in PostgREST.
Like passing the complete url through a header or an X-Request-Id using nginx $request_id.

Btw, I'll delete the placeholder Nginx snippet in http://postgrest.org/en/v5.2/admin.html#count-header-dos. Looks a bit sloppy for official docs.

Seems this can be done with a combination of https://serverfault.com/a/360151/468864 and https://stackoverflow.com/a/44537739/4692662.
Another choice is doing it with openresty https://stackoverflow.com/a/53086188/4692662. But I think we should stick to plain Nginx where possible.


fishPlugins.done: 1.16.3 -> 1.16.5

Motivation for this change
Update to latest version.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 140517 at 50c8c208 run on x86_64-linux 1

1 package built successfully:

fishPlugins.done

3 suggestions:

warning: missing-phase-hooks
configurePhase should probably contain runHook preConfigure and runHook postConfigure.
Near pkgs/shells/fish/plugins/build-fish-plugin.nix:37:10:
   |
37 |   inherit unpackPhase configurePhase buildPhase;
   |          ^

warning: missing-phase-hooks
checkPhase should probably contain runHook preCheck and runHook postCheck.
Near pkgs/shells/fish/plugins/build-fish-plugin.nix:69:3:
   |
69 |   checkPhase = ''
   |   ^

warning: missing-phase-hooks
buildPhase should probably contain runHook preBuild and runHook postBuild.
Near pkgs/shells/fish/plugins/build-fish-plugin.nix:37:10:
   |
37 |   inherit unpackPhase configurePhase buildPhase;
   |          ^

Result of nixpkgs-review pr 140517 run on x86_64-linux 1

1 package built:

fishPlugins.done


Errors when starting the emulator in Podman

Hi all,
Currently attempting to run this emulator on Podman desktop v 1.10.3 via Windows
After getting the containers up in podman and starting the run, the errors attached are received in the emulator logs:
PlatformNotSupportedException and NullReferenceException
Have attempted to uninstall and completely reinstall everything from podman through to the emulator. No success.
Unable to replicate the issue on another machine so far.
(sorry this thing won't let me input my screenshot of the errors)

@kacety123 ;
Can you please share the console logs using:
docker logs -f --tail 100 eventhubs-emulator

And file logs from the container at:
"/home/app/EmulatorLogs/*.log"

Also need few more details:
a) Windows Version
b) WSL Version
c) Emulator launch method.
If you are launching via the default bundling Windows script (LaunchEmulator.ps1) it requires updating few commands as:
Following docker compose commands change to podman compose commands :
docker compose -f $composeFile down
docker compose -f $composeFile up -d

podman compose -f $composeFile down
podman compose -f $composeFile up -d

With the above changes I was able to launch Emulator successfully:
![image](https://github.com/Azure/azure-event-hubs-emulator-installer/assets/121954204/6b035d3c-f955-41bc-a60b-0fb0dd64ddf2)

Hi yes so no success at getting the logs from the file but here is the information you were asking for:
Windows 11
Wsl version is 2
Launch method: docker
Logs from the emulator running:
[12:38:45 WRN] <Trc Id="60006" Ch="Debug" Lvl="Warning" Kw="1000000000011110" UTC="2024-06-27T12:38:45.748Z" Msg="An exception was handled at TenantCacheComponent.OnEndRefresh. Exception message ExceptionId: 4e48e70c-55c1-4a17-8431-c0c92acce328-System.NullReferenceException: Object reference not set to an instance of an object.
   at a.a.aaX.z(IAsyncResult)." /> [12:38:45 WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-06-27T12:38:45.785Z" Msg="Exception occurred while creating performance counters. Exception message &apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.
   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)
   at h.hM.ad()
   at h.hM.ab()
   at h.hh.A(PerformanceCounterLevel)&apos;." /> [12:38:46 WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-06-27T12:38:46.803Z" Msg="Exception occurred while creating performance counters. Exception message &apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.
   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)
   at h.hK.ab()
   at h.hh.A(PerformanceCounterLevel)&apos;." /> [12:38:46 WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-06-27T12:38:46.835Z" Msg="Exception occurred while creating performance counters. Exception message &apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.
   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)
   at h.hK.ab()
   at h.hh.A(PerformanceCounterLevel)&apos;." /> [12:38:46 WRN] <Trc Id="60006" Ch="Debug" Lvl="Warning" Kw="1000000000011110" UTC="2024-06-27T12:38:46.848Z" Msg="An exception was handled at TenantCacheComponent.OnEndRefresh. Exception message ExceptionId: 6ccf5f5f-2982-4f7f-af62-b362db018d3e-System.NullReferenceException: Object reference not set to an instance of an object.
   at a.a.aaX.z(IAsyncResult)." /> [12:38:46 WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-06-27T12:38:46.987Z" Msg="Exception occurred while creating performance counters. Exception message &apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.
   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)
   at h.hp.ac()
   at h.hp.ab()
   at h.hh.A(PerformanceCounterLevel)&apos;." />

@kacety123 ; PlatformNotSupportedException and NullReferenceException are internal warnings and should will not impact the Emulator runtime.
In the console logs you will get following (If you select the Emulator container in Podman):
info: a.a.aap[0]
      Emulator Service is Launching On Platform:CBL-Mariner/Linux,X64
info: a.a.aap[0]
      Instantiating service components
info: a.a.aap[0]
      Creating namespace, entities and consumer groups
info: a.a.aaW[0]
      Emulator is launching with config : {"LoggingConfig":{"Type":"File"},"NamespaceConfig":[{"Type":"EventHub","Name":"emulatorns1","Entities":[{"Name":"eh1","PartitionCount":2,"ConsumerGroups":[{"Name":"cg1"},{"Name":"$default"}]},{"Name":"eh2","PartitionCount":2,"ConsumerGroups":[{"Name":"cg2"},{"Name":"$default"}]}]}]}
info: a.a.aap[0]
      Emulator Service is Successfully Up! ; Use connection string : "Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;"

You can trying doing a Send-Receive with the bundled C# sample or any other official Event Hub SDK for other languages.

@kacety123 ; Additionally while launching via direct docker/podman compose , following environment variable needs to be specified either as Full Path or Relative to the compose.yaml.
   volumes:
      - "${CONFIG_PATH}:/Eventhubs_Emulator/ConfigFiles/Config.json"

If this is not specified Emulator defaults to its internal logging and entity configuration looks like is the case here.  When initialised correctly these warning log lines will not be pushed to console and useful info level logs will pop up around Emulator launch.

@kacety123 ; Any updates here ?

Closing due to inactivity. Validated that EH Emulator has no issues while launching with Podman. The errors mentioned above are warning level traces that ideally should not flow to console given Config.json is supplied to the Emulator. This does not impact any runtime operations with Emulator.


Implement a nullable annotation for fields and messages

Closes #24

I can't load these generated rules into the firestore emulator:
    Error: 3 INVALID_ARGUMENT: Error compiling rules:
    L14:101 Unexpected 'null'.

Should that be == null instead of is null?
I had a couple other observations after using this for a while:

I think what I really want is to be able to set this by type. In particular I'd like a file option to make dates and references nullable.
I'm working on an ORM to generate typesafe code in multiple languages (go, typescript, java) that enforces these schemas. As part of that I've been working on bazel macros for working with firestore rules and testing them using the firestore emulator. I was planning to release them with my ORM, but I'm wondering now if you'd want to put them here. I have:

firestore_rules_library and firestore_rules_binary for combining multiple files into a complete .rules file
firestore_rules_proto_library to generate the .rules file using this generator and put it into a firestore_rules_library
an example of starting and using the firestore emulator from a jasmine_node_test

It should be ==, thanks for catching
I'd welcome contributions for the per type config! It shouldn't be too bad.
RE: bazel rules, I think those specific rules would be great to have in this repo and I'd welcome those contributions!
I think as long these macros/rules are primatives and not specialized to the ORM, this repo is a good place for them.


Extending JSON for Query specification.

When key used in a kv storage is shaped as JSON string, then we need some additional support besides what is available in JSON spec

Min-Unbounded, to denote that we want to iterate from the first item.
Max-Unbounded, to denote that we want  to iterate till the last item.
-Infinity, to denote smallest number.
+Infinity, to denote largest number.
NaN, since it is included in IEEE 754 spec.
Missing, what is the sort position for missing in the supported list of types.

Terminator  byte =  0
TypeMissing byte = 11
TypeNull    byte = 12
TypeFalse   byte = 13
TypeTrue    byte = 14
TypeNumber  byte = 15
TypeString  byte = 16
TypeLength  byte = 17
TypeArray   byte = 18
TypeObj     byte = 19
TypeBinary  byte = 20
Creating a gap between Terminator and TypeNull, for more types that can be
included later on.

These representations can be added in collation binary format.
While some more analysis are required for JSON and CBOR.


Release Automation Tweaks

Task Description

[x] Improve pre-flight checks

[x] Check for changelog entry in readme.txt

[x] Fix bash commands
[ ] Documentation
[ ] Figure out how to trigger Firebase deployment for static assets (as per known limitation)
[ ] SVN user for deployments
[ ] Move environment protection to cover whole workflow, not just the WP.org deployment
[ ] (Post-)release version bumps on main

Some resources for pushing to main:
https://github.com/benjefferies/branch-protection-bot
https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token


Improve REST Path property for Path Params

When creating a rest method you can use {something} to indicate a path param. There is no way in the Path field to configure this and notice the existence of path params.

@hguerrero could you please elaborate/add screenshots ? I dont understand the problem

@davsclaus could we add more information to REST models in catalogue?

Yes the description in the model can be updated, then when you click that ? icon then it shows that.
Anyone is welcome to help write a better description we can use
https://github.com/apache/camel/blob/main/core/camel-core-model/src/main/java/org/apache/camel/model/rest/RestDefinition.java#L104

Captions come from Camel itself and if requires should be fixed there


[XLA:GPU][ROCm] Enabling amdgpu backend in XLA unit tests

This PR enables amdgpu XLA backend for unit tests, in preparation for setting up ROCm XLA community support builds.

@jerryyin Could you please check failed build errors? Thanks!

@gbaned Judging from the build failures: external/local_config_mlir/include/mlir/Dialect/QuantOps/UniformSupport.h:148:14: error: 'clamp' is not a member of 'std'
I don't think the failures have anything to do with this PR. std::clamp is a C++ 17 addition and shouldn't be used in that header file.


fix: use fc binary path

Current dependencies on/for this PR:

main

PR #26   👈

This stack of pull requests is managed by Graphite.


crash on GGML_ASSERT: 'rwkv.cpp/ggml/src/ggml.c:5316: ggml_can_repeat_rows(b, a)'

System: Linux x86_64, Ubuntu 22.04.3 LTS
python3 python/generate_completions.py RWKV-5-World-0.1B-v1-20230803-ctx4096-ggml.bin
System info: AVX=1 AVX2=1 AVX512=0 FMA=1 NEON=0 ARM_FMA=0 F16C=1 FP16_VA=0 WASM_SIMD=0 BLAS=0 SSE3=1 VSX=0
Loading RWKV model
GGML_ASSERT: rwkv.cpp/ggml/src/ggml.c:5316: ggml_can_repeat_rows(b, a)
Aborted (core dumped)

gdb
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350497344) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350497344) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737350497344) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737350497344, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7cca476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7cb07f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fffd30baea4 in ggml_add_impl (ctx=0x7fffd31d6908 <g_state+104>, a=0x7fffc03ffad0, b=0x7fffb8753e90, inplace=false)
    at /work/rwkv.cpp/ggml/src/ggml.c:5316
#6  0x00007fffd30bafdb in ggml_add (ctx=0x7fffd31d6908 <g_state+104>, a=0x7fffc03ffad0, b=0x7fffb8753e90)
    at /work/rwkv.cpp/ggml/src/ggml.c:5340
#7  0x00007fffd30f83c0 in rwkv_att_wkv (ctx=0x7fffd31d6908 <g_state+104>, att_time_first=0x7fffc03ffad0, att_time_decay=0x7fffc03ff950,
    k=0x7fffb8753e90, v=0x7fffb8753fe0, aa=@0x7fffffffcf20: 0x7fffb87486d0, bb=@0x7fffffffcf28: 0x7fffb8748820,
    pp=@0x7fffffffcf30: 0x7fffb8748970) at /work/rwkv.cpp/rwkv_graph.inc:132
#8  0x00007fffd30f86fc in rwkv_att (ctx=0x7fffd31d6908 <g_state+104>, x=0x7fffb8752ae0, layer=..., state=...)
    at /work/rwkv.cpp/rwkv_graph.inc:172
#9  0x00007fffd30f90e3 in rwkv_build_serial_graph (model=..., graph=...) at /work/rwkv.cpp/rwkv_graph.inc:274
#10 0x00007fffd30f9552 in rwkv_measure_and_build_serial_context (model=..., graph=...) at /work/rwkv.cpp/rwkv_graph.inc:320
#11 0x00007fffd30fa6db in rwkv_init_from_file (
    file_path=0x7fffd37e7770 "/work/models/RWKV-5-World-0.1B-v1-20230803-ctx4096-ggml.bin", n_threads=4)
    at /work/rwkv.cpp/rwkv.cpp:65
#12 0x00007ffff7083052 in ffi_call_unix64 () from /.conda/envs/pytorch/lib/python3.11/lib-dynload/../../libffi.so.8
#13 0x00007ffff7081925 in ffi_call_int () from /.conda/envs/pytorch/lib/python3.11/lib-dynload/../../libffi.so.8
#14 0x00007ffff708206e in ffi_call () from /.conda/envs/pytorch/lib/python3.11/lib-dynload/../../libffi.so.8
#15 0x00007ffff70932e4 in _call_function_pointer (argtypecount=<optimized out>, argcount=2, resmem=0x7fffffffd1a0, restype=<optimized out>,
    atypes=<optimized out>, avalues=<optimized out>, pProc=0x7fffd30fa471 <rwkv_init_from_file(char const*, uint32_t)>, flags=4353)
    at /usr/local/src/conda/python-3.11.4/Modules/_ctypes/callproc.c:923

llama.cpp works fine.

H!

RWKV-5-World-0.1B-v1-20230803-ctx4096-ggml.bin

RWKV 5 is not supported yet. I have plans to support it, but can't say when it will be done.


Why do snippets with isFileTemplate=true also show in the middle of the file?

Type: Bug
Doesn't make a lot of sense.  I had hoped isFileTemplate=true would solve my bug (https://github.com/orgs/Azure/projects/115/views/39?pane=issue&itemId=28512515) where a snippet intended for "bicepconfig.json" files shows up in the middle of all JSONC files.  If it only showed up in an empty JSONC file that would be an improvement.
I don't understand why an isFileTemplate=true template would show up when the file is no longer empty.
REPRO:

install ms-azuretools.vscode-bicep extension
open vscode's settings.json file
put cursor onto an empty line in the middle of the file
CTRL+SPACE
!) the "Default Bicep Configuration" snippet shows up.  This happens even if I set isFileTemplate=true on the snippet.
DESIRED:  I really want this snippet to only show up for files named "bicepconfig.json", but there seems to be no way to do that.  If isFileTemplate=true restricted it to empty JSONC files, that would at least be a big improvement.  Thx.

VS Code version: Code - Insiders 1.80.0-insider (Universal) (8c85702bca51b08005ada7fc4eaf52a61623b2b6, 2023-06-20T05:36:10.312Z)
OS version: Darwin x64 22.5.0
Modes:

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2600)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
2, 3, 3

Memory (System)
16.00GB (0.03GB free)

Process Argv
--enable-proposed-api ms-azuretools.vscode-bicep --crash-reporter-id 39d91d6b-0aa9-4ae4-8826-427cf9cf9ecb

Screen Reader
yes

VM
0%

Extensions (40)

Extension
Author (truncated)
Version

vscode-json
and
1.5.2

vscode-eslint
dba
2.4.0

gitlens
eam
14.0.1

EditorConfig
Edi
0.16.4

prettier-vscode
esb
9.15.0

copilot
Git
1.90.174

vscode-github-actions
git
0.25.8

vscode-pull-request-github
Git
0.67.2023062009

debug-visualizer
hed
2.4.0

json-escaper
jos
1.1.2

package-json-dependencies-navigation
mam
0.1.8

vscode-antlr4
mik
2.3.1

vscode-azureresourcegroups
ms-
0.7.5

vscode-bicep
ms-
0.18.4

vscode-cosmosdb
ms-
0.19.4

vscode-docker
ms-
1.25.1

extension-manifest-editor
ms-
0.1.5

csharp
ms-
1.25.9

vscode-dotnet-runtime
ms-
1.6.0

remote-containers
ms-
0.296.0

remote-ssh
ms-
0.102.0

remote-ssh-edit
ms-
0.86.0

remote-wsl
ms-
0.79.5

vscode-remote-extensionpack
ms-
0.24.0

azure-account
ms-
0.11.5

powershell
ms-
2023.6.0

remote-explorer
ms-
0.4.0

remote-server
ms-
1.2.1

vscode-selfhost-test-provider
ms-
0.3.15

azurerm-vscode-tools
msa
0.15.10

json-parse-stringify
nex
1.0.2

vscode-json-editor
nic
0.3.0

vscode-yaml
red
1.13.0

vscode-sort-json
ric
1.20.0

vscode-javascript-booster
sbu
14.0.1

scope-info
sie
0.2.0

tmcolor
sle
0.0.1

sort-lines
Tyr
1.10.1

win-ca
uko
3.5.0

vscode-todo-highlight
way
1.0.5

A/B Experiments
vsliv695:30137379
vsins829:30139715
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256197
vslsvsres303:30308271
pythontb:30258533
pythonptprofiler:30281269
vshan820:30294714
vscod805cf:30301675
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30404738
vsclangdf:30492506
c4g48928:30535728
dsvsc012:30540252
pynewext54:30618038
pylantcb52:30590116
pyind779:30611226
pythonsymbol12:30651887
showlangstatbar:30737417
pythonms35:30671666
03d35959:30757351
ecj1e332:30687743
pythonfmttext:30716741
pythoncmvfstr:30726892
fixshowwlkth:30771523
hideindicator:30766887
pythongtdpath:30726887
gsof1:30771514
dh2dc718:30770000
pythonidxpt:30768918
pythondjangots:30768917
pythonnocebcf:30766856
copilotsettingt:30767686
e537b577:30772214

If I remember correctly we did that on purpose because not all file templates aren't applicable in non empty files, e.g you might have a file template to add a new class or (global) function and that's always applicable.

Perhaps isFileTemplate could accept "never", "only" and "also", with "never" being equivalent to false, "also" to true, and "only" indicating that the snippet should only be offered when creating a file from a template.

should only be offered when creating a file from a template.

I'd say this should be "when creating a file from a template or in an empty file".  Creating from a template right now is kind of hidden, and lots of people (including myself) are used to creating a new file, setting language, and typing.

not all file templates aren't applicable in non empty files

I would argue that if a file template is not applicable to the entire file, then it isn't really a file template then

@StephenWeatherford a workaround could be
if you are in control of the schema
you can add a "defaultSnippets" section at the root level

only shows up at root level (eg empty file)

package.json
"contributes": {
	"jsonValidation": [
		{
			"fileMatch": [
				"bicepconfig.json"
			],
			"url": "./bicepconfig.schema.json"
		}
	]
}

@jrieken That doesn't make much sense given the documentation:
https://code.visualstudio.com/docs/editor/userdefinedsnippets#_file-template-snippets
You can add the isFileTemplate attribute to your snippet's definition if the snippet is intended to populate or replace a file's contents. File template snippets are displayed in a dropdown when you run the Snippets: Populate File from Snippet command in a new or existing file.

@RedCMD
I would argue that if a file template is not applicable to the entire file, then it isn't really a file template then

Exactly.
FYI, I do not have control of the schema.


Add a raw meta.yaml for conda packaging.

I (Feth) am not the author/maintainer of python-slugify
I am willing to transfer ownership

This will allow 'conda install python-slugify'.
Result is at https://anaconda.org/feth/python-slugify

Coverage remained the same at 94.048% when pulling 165ded9273ec7e85e3d5a00552281fccf8ab6e54 on feth:master into 6472f8fae4088f323a69ef5086f0f97268be88f8 on un33k:master.

Coverage remained the same at 94.048% when pulling c5988c1577339575ffaddc3c6b5c4459a35bcb4b on feth:master into 6472f8fae4088f323a69ef5086f0f97268be88f8 on un33k:master.

No plan to add condo install support at this time.

Hello,
sorry to read this as there's actually only a feature added (that I use, so I can say that it works as of now).
I believe you fear that some irrelevant issues from conda used may add noise, or that you would have trouble to reproduce them and I can understand this.
With my limited knowledge of this ecosystem I have no reason to believe that your very useful package works differently under conda.
Feth


Reimplement Outbox leveraging Rebus implementation

The default implementation of Outbox isn't well documented and only supports SqlServer: https://github.com/rebus-org/Rebus/issues/819 + https://github.com/rebus-org/Rebus.SqlServer/tree/master/Rebus.SqlServer/SqlServer/Outbox


A string containing only numbers will be type cast into a number by ConvertToJson

Using a dictonary as body for a post request resulted in a type mismatch whenever the string only contained numbers.
Looking into the implementation reveals that ConvertToJson is casting a string containing only numbers into a json number.

Case VBA.vbString
        ' String (or large number encoded as string)
        If Not JsonOptions.UseDoubleForLargeNumbers And json_StringIsLargeNumber(JsonValue) Then
            ConvertToJson = JsonValue
        Else
            ConvertToJson = """" & json_Encode(JsonValue) & """"
        End If

In our case this is explicitly not wanted. Having the library silently changing the type feels actually like a bug.

Could you paste some code?
Public Sub TestNumbers()
    Dim dict As New Dictionary
    Dim aNumber As Double
    aNumber =<PHONE_NUMBER>.23423
    dict.Add "foo", 145
    dict.Add "bar", aNumber
    Debug.Print ConvertToJson(dict)
End Sub

'{"foo":145,"bar":3423423423.23423}'

This is intentional (although an iffy decision) for handling BigInts so that they don't get truncated. You can turn it off with JsonOptions.UseDoubleForLargeNumbers = True, but I have some plans for the next version for VBA-JSON to alleviate this issue.

@timhall  I understand the necessity for bigint handling.
Although the current implementation results in an unexpected behaviour modifying variable types.
In addition UseDoubleForLargeNumbers = True would have the effect wished for, although my understanding is, that the flag changes parsing json as well.
Summed up to me it feels 'wrong' to set a flag, intended to handle bigint's a different way, to get the library to submit any passed 'number only' string correctly.


Support for subscription-manager config options #339

Added support for subscription-manager config options.
Just add to your yaml:
...
rhn_user: "joe"
rhn_config_options: "--server.proxy_hostname=proxy.example.com --server.proxy_port=8080"
...

This looks good to me and doesn't break anything, but I'd appreciate @markllama's thoughts. Mark, you know RHN better than I do. Are you okay with this going in?

@tomassedovic @markllama does it need any refactoring?

@markllama @tomassedovic ping


License

Hi,
Nice project!
I noticed there is no LICENSE file in the repo. Could you add one?

Thank you! I added the license files (GNU v3 for the modified rikaichamp, MIT for my code base).


[FR]: Add Additional Languages

Describe your idea
Some of our Discord user discussed that they would like to help us expand the pool of planned supported languages of Stereum
Currently these are the following languages that we got offered:

[x] Czech
[x] Vietnamese

Considered Alternatives?
No alternative - they deserve to be added as well - if we could, we would add them all.
Additional Context
--
Acknowledgment

[X] I looked through the feature requests/enhancements planned for this project before submitting a new idea

✅


Postman requests take into account K6 param options

This is a PR to provide support in the Postman request for passing K6 params that are globally defined.
https://k6.io/docs/javascript-api/k6-http/params
Example
export default function () {
  let params = {
    cookies: { my_cookie: 'value' },
    headers: { 'X-MyHeader': 'k6test' },
    redirects: 5,
    tags: { k6test: 'yes' },
  };
  let res = http.get('https://k6.io', params);
}

It seemed that the Postman request function does not uses all the "options", when executing the http.request via K6.
Example script.js file
// Auto-generated by the Load Impact converter

import "./libs/shim/core.js";
import { group } from "k6";
import "./requests/Lists/Get-all-list-configurations.js";

export let options = { maxRedirects: 4 , headers: { 'Content-Type': 'application/json' } };

const Request = Symbol.for("request");
postman[Symbol.for("initial")]({
  options,
  collection: {
    baseUrl: "https://example.com/api"
  },
  environment: {
    baseUrl: "https://example.com/api"
  }
});

...

Which results in the following K6 call
INFO[0006] Request:
GET /api/lists/v2?$top=100&$skip=0&$count=false HTTP/1.1
User-Agent: k6/0.29.0 (https://k6.io/)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip

The PR adds the passing of let options configurations, when making the K6 requests.
export let options = { maxRedirects: 4 , headers: { 'Content-Type': 'application/json' } };
Which results in
INFO[0006] Request:
GET /api/lists/v2?$top=100&$skip=0&$count=false HTTP/1.1
User-Agent: k6/0.29.0 (https://k6.io/)
Content-Type: application/json
Accept-Encoding: gzip

This PR would allow the possibility to leverage any K6 Params https://k6.io/docs/javascript-api/k6-http/params to be used when executing the Postman Request via K6.

I need to test this before merging, but from eyeing it this looks great. Good job, @thim81! 👏🏼

Thanks a bunch for your contribution! 🎉


IIFE causing AST to not visit it

Describe the bug
Hey there, I'm porting a babel plugin that extracts strings from function calls (gettext/localization stuff).
With babel, this snippet would have received 3 visits (2 for translate and one for the IIFE) to CallExpr
const a = translate("transform");
const b = (() => translate("transform"))();

https://astexplorer.net/#/gist/87057f51b642e6cbd25df31c5c1c3855/a5c5e4dc30c5229db8e8d627ed9639945a106781
But the same is not happening with SWC, we only get 2 visits, the inner CallExpr is ignored.
It seems like a bug to me, I tried to have workarounds, but it can get really ugly, I'm wondering if that's expected or not.
Input code
use swc_core::ecma::{
    transforms::testing::test,
    visit::{as_folder, VisitMut},
};
use swc_core::ecma::ast::CallExpr;

pub struct TransformVisitor;

impl VisitMut for TransformVisitor {
    fn visit_mut_call_expr(&mut self, n: &mut CallExpr) {
        dbg!(&n.callee);
    }
}

test!(
    Default::default(),
    |_| as_folder(TransformVisitor),
    boo,
    // Input codes
    r#"const a = translate("transform"); const b = (() => translate("transform"))();"#,
    // Output codes after transformed with plugin
    r#"const a = translate("transform"); const b = (() => translate("transform"))();"#
);

Config
No response
Playground link
No response
Expected behavior
CallExpr is visited 3 times
Actual behavior
No response
Version
0.40.31
Additional context
No response

You should read plugin docs

@kdy1 I couldn't find anything in the docs that helped me, I also searched for issues and discussions here on Git, are you saying that this is not a bug from your perspective?


Cagra support & update raft version

issue: #208

kind/feature

/run-e2e

/run-e2e

/run-e2e

/lgtm

/lgtm
given the assumption that a large topk for ivfpq will be tested and added :)


Known issue: station details outdated

The details of the stations are derived from a file, see link below
This file is not maintained and has not been updated since years, as a result your stations may show an old name or no name at all.
dd. 30/6/2022 there seems to be no other centralized solution providing the same data in manageable format.
This remains open issue until such a solution is provided by the French government or if someone (else) takes provides a workaround.
Suggestions are welcome
https://static.data.gouv.fr/resources/prix-des-carburants-en-france/20181117-111538/active-stations.csv

Could you display the station address instead of his ID when the name is unavaliable ?
And about the entity ID, maybe keep the station ID in case the name change at some point ?

You can also fetch data like them and store them in this repo;
https://github.com/floman321/prixcarburants/tree/master/data/listestations

@Zehir

showing the address: the address is part of the sensor so displaying that is a matter of configuring the output. This is imo not something that this integration should provide on top.
On the link, I verified the API that @roumano sent on a station close to me. The API does not show the station name, just address and services. The github link is showing more and FINALLY the name of my station. However, I am worried about the validity of this data, i.e. is this complete and will it be maintained? My questions towards extending the API provider run dead as they usually do with gouv stuff...but it should actually be the single source of truth, not something that depends on the work of a single person like floman321

Floman just scrape the stations names, the related code is provided. I think it's better than a file from 2016

Anything is better than that file but this is what the original implementation was based on.
If you would ask me how to progress, I would say that this fork needs to incorporate the scrape results, to not depend on someone else...but it is quite a bit of work I guess, so tbd by Matthieu / @Aohzan

If I understand, we should migrate to this API. :+1:
But to get the station name, we have to use something like this, which is webscraping :nauseated_face:

So, from what I know and have read above only (!) the webscraping provides the station name, currently downloaded in floman321 github. The api does not bring any additional important (!) info ... for as far as I can see
I also donot know why that site's data is not available as api but I could not find it and neither can others, leading to the webscrape.

And what about using an other api like this ? https://api.zagaz.com/

As it's a different data source, which need a api key from the users, I think it has to be a new separate integration

Otherwise we put the address in the station name and we let the user renames them ?

Why would this be needed, the user can select its choice from the attributes? I see that you might liek this but I am not sure if this serves all others. The examples of the tables I added also show the address... donot see the added value

version 3.0.0 fix this


Two ways to supply client id.

Added -C,--client-id option to supply twitch application client ID on demand.
Environment variable TWITCH_CLIENT_ID can also set the client ID, and takes precedence.
Added long alias --color for the color option.
I'd also pulled your current client ID from the code. IMO, it should be regenerated on the twitch side and kept secret.

Thank you very much for the patch, I will have a look at it today.
The Client ID does not have to be kept secret, that's even impossible for applications like this.

As a last little nit, I'd appriciate if you could minimize the undescribed side effects of the patch, namely adding line breaks and chanign quotes in else unaffected code.

Oh, and please document the addition in the README.md.

Superseded by MR #4


fix(types): make request & response generic and remove express dependency

So as ya'll might know, http-proxy-middleware is not coupled with express or any other third-party library. But still the types reflect that.
A few problems that pop because of that...

http.createServer(createProxyMiddleware({}))

...won't compile when it should

http.createServer(createProxyMiddleware({ onError: (_, request) => { console.log(request.originalUrl.toString()) } }))

...shouldn't compile when it does and would create a runtime error "Cannot read property 'toString' of undefined" because originalUrl is express-specific

const myFavLibraryUse = (handler: (request: http.IncomingMessage, response: http.ServerResponse & { doMagic: () => {} }) => void) => {}
 myFavLibraryUse(createProxyMiddleware({ onError: (_, __, res) => res.doMagic() }))

...won't compile when it should

A project using http-proxy-middleware but not having express installed won't compile unless with --skipLibChecks because types.ts imports express

The solution to all these problems in making request and response generic which consequently would remove express dependency
This will be a breaking change for the people who...

have written let middleware = createProxyMiddleware({ ... }) (instead of directly passing to use so contextual inference works) AND
and use request, response in any of the options AND
access library exclusive properties or methods on request and response

So it'll be a breaking change only if ALL these three are fulfilled, otherwise there's no problems.
The easiest way to migrate is to directly pass to use. If for any reason the users can't do that, just annotate request and/or response whatever is being used with the library specific types.
Aside: It's my first time doing some server side work with TypeScript and I'm not using express and I see this pattern of libraries having their types depend on express, and not adhering to their source code (that doesn't require express exclusive methods & properties). Like serve-static that I recently fixed. It's probably because it's easy to use express types so it seamlessly works with express (as it's popular) and get done with it, but let's change that ;)

Umm is the CI failing because of next && next(err) or next && next() instead of unchecked calls to next? Because that's the only change in runtime code 🤔

Coverage increased (+5.7%) to 93.989% when pulling 9b8a5a26e5cc4841e585fb7441e28907b2f719b8 on devanshj:fix-types-remove-express into f212a047ef45898ce6053b622cada4b0617b9ddf on chimurai:master.

aside from internalizing the types, another possible approach is to add express as a peer dependency so that the express types can be reused.

I like the intent of this PR, but it does a few things that aren't ideal.

@ts-ignores are a bit of a code smell, imho
next. it is a middleware library, thus next is a key param in middleware impls. i think it's fine to keep it in and require it. if this m/w is used as a final handler, folks can always drop in a noop

take a peek @ #721. i'm viewing all of the TS complaint issues, ensuring that #721 appeases all complaints/use-cases, and at this point i think it does, with this one perhaps being the exception, due to the feature request of dropping next

@ts-ignores are a bit of a code smell, imho

I don't know what @ts-ignores you are talking about, the @ts-expect-error are for testing types.

next [...] require it

I just kept the behavior as it is. What you're suggesting is a breaking change afaict, I don't have any opinions on that.

#721

I haven't checked it but as long as it solves the problem this PR solves I'm good

Fixed in https://github.com/chimurai/http-proxy-middleware/pull/730
Removal of @types/express will be investigated in separate PR


Reference or citation to be added for paper

Hi Authors/developers,
I used this wonderful package for my research. Do you have any citations/references for the package that I can add to my paper?
Thank you again.

Hi Hellorp1990, there had been a few articles on medium about some of verstacks modules, here are the links:
https://danilzherebtsov.medium.com/parallelise-like-a-boss-with-a-single-line-of-code-in-python-30af0d640511
https://danilzherebtsov.medium.com/automatic-feature-selection-in-python-f72ec69215fe
https://danilzherebtsov.medium.com/meet-stacker-automated-ensemble-creation-library-abcad0648a5d
https://danilzherebtsov.medium.com/ultimate-python-dateparser-with-holidays-paydays-and-all-the-good-stuff-fffe270a236
https://danilzherebtsov.medium.com/effortlessly-tune-lgbm-with-optuna-49de040d0784
https://danilzherebtsov.medium.com/advanced-categoric-encoding-like-a-pro-onehot-meantarget-woe-frequency-factorization-99dc69477bf3
https://medium.com/swlh/impute-missing-values-the-right-way-c63735fccccd
https://danilzherebtsov.medium.com/continuous-data-stratification-c121fc91964b

Is this what you are looking for?
And which of the modules have you been using in your research?

@DanilZherebtsov  Thank you for the links. I was looking for any published papers. But medium links should be okay. I utilized NanImputer for the current project.

@hellorp1990 BTW next generation NaNImputer is getting ready for release. It will be much faster than the current version.
If possible, please share a link to you paper when published!

@DanilZherebtsov Thanks for letting me know. Hopefully, the new version will be helpful for my other projects where there are several missing data and make it faster.
I am submitting my current study now. I will share the link once accepted/published.

@hellorp1990, the new and upgraded version of NaNImputer had been released. It works much faster, has a simpler architecture and is more reliable. You can upgrade to the new version of verstack by $ pip install --upgrade verstack.
The previous version has been kept since it has more configurability under the name NaNImputerLegacy. It is all reflected in the documentation.
Did you get your paper published?


Check in Cargo.lock

With the stabilization of Cargo.lock file publishing in rust-lang/cargo#7026, lets restart the discussion from #148 about adding Cargo.lock to the repo.
My two cents: not having Cargo.lock checked in is a huge road block to using svd2rust in production.
Backgroud: we are running svd2rust on every CI build. By not locking svd2rust, there is a chance that the final artifact is not reproducible if a package dependent on by svd2rust fails to adhere to semver. More realistically, such an update breaks the build, which is completely unacceptable.

The first stable version of Cargo that supports publishing lock-files is 1.37.

I disagree with your reasoning: Semver has proven to provide pretty good stability that API breakage does pretty much never happens.
On the other hand people are continuously improving their crates in a compatible manner, ranging from dependency cutting over optimisation all the way to fixing bugs detected by newer Rust compiler versions or forward compatibility lints that it actually is very desirable to automatically receive those updates without having to intervene manually, which is a cumbersome and very annoying maintenance task.
Indeed as you mentioned the purpose of Cargo.lock is to provide reproducible builds however I don't see much value in rebuilding svd2rust continuously. If you want to have a defined and stable version of svd2rust, why not compile a specific version, execute your tests and then re-use that pre-built artefact as long as you like. Alternatively you could also use the new cargo ability to download all dependencies and do an offline build, then you'll also get the exact same dependencies every time.
IMHO the far bigger hazard here is that svd2rust changes in an incompatible way...

I agree that semver generally works quite well, and that picking up the latest version of compatible crates is generally desirable for end users. Note that this won't change when including Cargo.lock files in published releases: Cargo will normally ignore the lock file, unless install --locked is used.
Regardless of if we would rebuild svd2rust on every build or if we would distribute an artifact, we would want to be able to 1. rebuild the artifact to prove it wasn't tempered with and 2. be able to apply fixes without having to trace back versions of dependencies that allowed the project to build.
Furthermore, it turns out that correctly applying semver is quite tricky. We've had breakages specifically due to semver when the edition field got added to Cargo.toml. Several crates released minor or patch version updates that added edition = 2015 to their toml file, which our older version of Cargo didn't understand. To reproduce, try building svd2rust v0.14.0 using Rust version 1.29.1.
I guess my question is: why not check in Cargo.lock?

I guess my question is: why not check in Cargo.lock?

Because it requires a ton of manual work to maintain.
I don't really see why we are supposed to do the pinning upstream when you can easily do it yourself. If you're so keen on building exact versions anyway you're most likely doing your own vetting/verification anyway so why not simply maintain your own fork with a Cargo.lock.

I don't think this is the way to go. If something breaks semver, we deal with it as fast as we can. But enforcing Cargo.lock requires too much work to keep up to date and provides negligible improvements.
If you want to make your CI work reproducibly, I'd say you should check in a Cargo.lock besides Cargo.toml in your CI source/builder.

This is a problem for packaging maintainers.  svd2rust is not reproducible without Cargo.lock.  Reproducible builds are required to package svd2rust on some distros.  For example, NixOS maintains an out-of-tree Cargo.lock to ensure builds are reproducible:

https://github.com/NixOS/nixpkgs/blob/3b6c3bee9174dfe56fd0e586449457467abe7116/pkgs/development/tools/rust/svd2rust/cargo-lock.patch
https://github.com/NixOS/nixpkgs/blob/3b6c3bee9174dfe56fd0e586449457467abe7116/pkgs/development/tools/rust/svd2rust/default.nix

My opinion is that reproducible builds add value, and requiring each user who desire reproducible builds to maintain their own Cargo.lock is of negative value.
This has all been discussed before, but there have been new developments since this issue was opened:

Dependabot has become a mature tool for automatically updating rust dependencies.
The svd2rust MSRV (1.46) supports publishing lock files.

Do either of these recent developments change the discussion / decision?

The svd2rust MSRV (1.46) supports publishing lock files.

That makes it even worse. Now we not only have to maintain the Cargo.lock but also make frequent releases for it to actually take effect.

Do either of these recent developments change the discussion / decision?

I'm not terribly opposed to having a Cargo.lock iff there was a reliable volunteer keeping the whole thing going. 😉 I'm not going to hold my breath for someone showing up though...

Lock files only matter for published binary crates when using cargo install --locked
I think adding a lockfile is fine if we making at least one ci check to use --locked
It doesnt need to be updated frequently imo.

No, if Cargo.lock is addded added, the whole CI needs to use it.

Ah yes, I was thinking that new features in respect to semver are additive, and a single check would catch when features are missing, but that does not work if the change is not in this particular check, eg we now parse x properly in the xml library, here is the test for it.

A lockfile has been added in #741