[0.9.1] Silence on Proxied Alertmanagers returns 404

Hi,
I configured that requests from browsers to our Alertmanagers will be proxied via unsee. But when looking at the developer console when trying to create a new silence in the UI, i see that the POST request to the proxy-endpoint returns a 404. We are using unsee 0.9.1.
request:
POST https://unsee.mydomain.com/proxy/alertmanager/test/api/v1/silences 404

snippet form the config:
...
servers:
  - name: test
    uri<EMAIL_ADDRESS>    timeout: 30s
    proxy: true
  ...
  listen:
    address: "<IP_ADDRESS>"
    port: 8080 # port in Docker container (we are running unsee in k8s)
    prefix: /
  ...

When triggering the silencing requests, i can see the following debug log:
time="2018-04-12T15:50:02Z" level=debug msg="[test] Proxy request for /api/v1/silences"

Any idea what causes the issue or how i can debug this?
Thx a lot,
Sven

POST https://unsee.mydomain.com/proxy/alertmanager/staging/api/v1/silences 404
That's not the correct path, see https://github.com/cloudflare/unsee/blob/master/proxy.go#L17, it's proxy/alertmanager/${name}, so the URI should be https://unsee.mydomain.com/proxy/alertmanager/test/api/v1/silences, not sure why you're seeing errors with staging instead of test. Likely you have more than one alertmanager server configured in unsee, can you post the config part for staging if that's correct?
time="2018-04-12T15:50:02Z" level=debug msg="[test] Proxy request for /api/v1/silences" that seems to suggest that requests for test alertmanager are proxied.
Do you have a single instance of unsee or do you have multiple each for a different alertmanager server?

Hi,
Sorry, that was only a copy/paste error in this ticket. I replaced "staging" with test in my example but forgot the part you've mentioned. i adjusted it now in my question above.

yes, we have multiple Alertmanager defined in the configuration, since all Alertmanager are behind Nginx with Basic Auth and therefore username/password needs to be part of the url
we are only running a single unsee instance

Thx Sven
The complete configuration looks like this (credentials/domains replaced by random strings)
alertmanager:
  interval: 60s
  servers:
    - name: test
      uri<EMAIL_ADDRESS>      timeout: 30s
      proxy: true
    - name: production-1
      uri<EMAIL_ADDRESS>      timeout: 30s
      proxy: true
    - name: production-2
      uri<EMAIL_ADDRESS>      timeout: 30s
      proxy: true
    - name: misc
      uri<EMAIL_ADDRESS>      timeout: 30s
      proxy: true
    - name: hosting-2
      uri<EMAIL_ADDRESS>      timeout: 30s
      proxy: true
    - name: hosting-1
      uri<EMAIL_ADDRESS>      timeout: 30s
      proxy: true
annotations:
  default:
    hidden: false
  hidden: []
  visible: []
debug: true
filters:
  default: []
labels:
  color:
    static:
      - job
    unique:
      - cluster
      - instance
      - "@receiver"
  keep: []
  strip: []
listen:
  address: "<IP_ADDRESS>"
  port: 8080
  prefix: /
log:
  config: true
  level: debug
jira:
  - regex: OPS-[0-9]+
    uri: https://jira.example.com
receivers:
  keep: []
  strip: []

I'm having trouble reproducing this with my setup. You have log:debug so you should see:
[GIN-debug] POST   /proxy/alertmanager/test/api/v1/silences --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (5 handlers)
[GIN-debug] DELETE /proxy/alertmanager/test/api/v1/silence/*id --> github.com/cloudflare/unsee/vendor/github.com/gin-gonic/gin.WrapH.func1 (5 handlers)

in logs on startup, can you verify if that's there?
Also what version of alertmanager are you using with test instance? Can you try to:
curl -XPOST<EMAIL_ADDRESS>
?
time="2018-04-12T15:50:02Z" level=debug msg="[test] Proxy request for /api/v1/silences" seems to suggest that request is send upstream, it's just that alertmanager1.mydomain.com/api/v1/silences returns 404 for some reason

we are running Alertmanager version 0.9.1
method POSTreturns an error

{"status":"error","errorType":"bad_data","error":"EOF"}

method GET works

{"status":"success","data":[...]}

logged events during start

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

time="2018-04-13T19:33:17Z" level=info msg="reqCnt registered."
time="2018-04-13T19:33:17Z" level=info msg="reqDur registered."
time="2018-04-13T19:33:17Z" level=info msg="reqSz registered."
time="2018-04-13T19:33:17Z" level=info msg="resSz registered."

-- Sven

In front of each alertmanager service, we have a nginx as reverse proxy running (for Basic Auth). All components are deployed in a k8s cluster.
Here is the used nginx config:
geo $google_lb {
    default 0;
    <IP_ADDRESS>/8 1;
}

map "$google_lb:$http_x_forwarded_for" $lb {
  default 0;
  "1:" 1;
}

server {

    listen 80;
    server_name alertmanager1.mydomain.com;

    if ( $http_x_forwarded_proto = "http" ) {
        return 301 https://$host$request_uri;
    }

    location /health {
        if ( $lb ) {
                rewrite ^/(.*) /\#/status break ;
                proxy_pass http://alertmanager1;
        }
    }

    location / {
        auth_basic "Restricted Content";
        auth_basic_user_file /etc/nginx/auth/htpasswd;
        proxy_pass http://alertmanager1;
    }
}

Using the Alertmanager API directly works:
curl<EMAIL_ADDRESS>\
-X POST --data @- << EOF
{
  "matchers": [
  ...
  ],
  "startsAt": "2018-04-14T10:35:41.436Z",
  "endsAt": "2018-04-14T11:35:41.722Z",
  "createdBy": "...",
  "comment": "test"
}
EOF
{"status":"success","data":{"silenceId":"a09f4706-6129-425c-b773-f071e126bf42"}}

Does the proxy maybe have issues when the username/password is part of the proxied url?
-- Sven

when i try the exact same request from unsee via curl (copied from firefox console):
curl -v 'https://unsee.mydomain.com/proxy/alertmanager/test/api/v1/silences' -H 'Accept: application/json, text/javascript, */*; q=0.01' --compressed -H 'Accept-Language: de,en-US;q=0.7,en;q=0.3' -H 'Connection: keep-alive' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Cookie: optimizelySegments=...; optimizelyEndUserId=...; optimizelyBuckets=...; _ga=...; amplitude_idmydomain.com=...; intercom-lou-fnb5rw5o=...; tk_tc=...; GCP_IAAP_XSRF_NONCE=...; GCP_IAAP_AUTH_TOKEN=...; _gid=...' -H 'Host: unsee.mydomain.com' -H 'Referer: https://unsee.mydomain.com/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0' -H 'X-Requested-With: XMLHttpRequest' --data ''

*   Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to unsee.mydomain.com (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.mydomain.com
*  start date: Oct 19 00:00:00 2017 GMT
*  expire date: Dec 18 23:59:59 2019 GMT
*  subjectAltName: host "unsee.mydomain.com" matched cert's "*.mydomain.com"
*  issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7feb06006000)
> POST /proxy/alertmanager/staging/api/v1/silences HTTP/2
> Host: unsee.mydomain.com
> Accept-Encoding: deflate, gzip
> Accept: application/json, text/javascript, */*; q=0.01
> Accept-Language: de,en-US;q=0.7,en;q=0.3
> Connection: keep-alive
> Content-Type: application/x-www-form-urlencoded; charset=UTF-8
> Cookie: optimizelySegments=...; optimizelyEndUserId=...; optimizelyBuckets=%7B%7D; _ga=...; amplitude_idmydomain.com=eyJkZXZpY2VJZCI6Ijc5Y2Q2NTM0LTkyMDQtNGQzMy05MDRiLTRhZDRkYjljMThhNlIiLCJ1c2VySWQiOiJmNWMwMTIwNy0wNjQ4LTQ5NDgtOWI1Ny1kNmYyMmViZWZlZGQiLCJvcHRPdXQiOmZhbHNlLCJzZXNzaW9uSWQiOjE0NzkxMzU2Mjc2NzEsImxhc3RFdmVudFRpbWUiOjE0NzkxMzg2MTkxMTUsImV2ZW50SWQiOjI5NSwiaWRlbnRpZnlJZCI6ODMsInNlcXVlbmNlTnVtYmVyIjozNzh9; intercom-lou-fnb5rw5o=1; tk_tc=1523532035454; GCP_IAAP_XSRF_NONCE=...; GCP_IAAP_AUTH_TOKEN=...; _gid=...
> Referer: https://unsee.mydomain.com/
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
> X-Requested-With: XMLHttpRequest
> Content-Length: 0
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< alt-svc: clear
< alt-svc: clear
< content-encoding: gzip
< content-type: text/plain; charset=utf-8
< date: Sat, 14 Apr 2018 10:50:15 GMT
< vary: Accept-Encoding
< content-length: 45
< via: 1.1 google, 1.1 google
* HTTP error before end of send, stop sending
<
* Connection #0 to host unsee.mydomain.com left intact
default backend - 404

The unsee service is running behind Google IAP (https://cloud.google.com/iap/docs/?hl=en), looks like the XMR request from the client fails for some reasons i don't understand yet.

Ok, i turned off Google IAP in front off unsee service but still the same issue ("default backend - 404"). Must be something related to the request done from the client (browser) and google ingress/loadbalancer.

@prymitive Ok, i think i found an issue. Looks like the unsee proxy thingy somehow sets a wrong hostname when proxying the requests.
Example:
curl -v 'https://unsee.mydomain.com/proxy/alertmanager/test/api/v1/silences' -H 'Accept: application/json, text/javascript, */*; q=0.01' --compressed -H 'Accept-Language: de,en-US;q=0.7,en;q=0.3' -H 'Connection: keep-alive' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Cookie: optimizelySegments=...; optimizelyEndUserId=...; optimizelyBuckets=...; _ga=...; amplitude_idmydomain.com=...; intercom-lou-fnb5rw5o=...; tk_tc=...; GCP_IAAP_XSRF_NONCE=...; GCP_IAAP_AUTH_TOKEN=...; _gid=...' -H 'Host: unsee.mydomain.com' -H 'Referer: https://unsee.mydomain.com/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0' -H 'X-Requested-With: XMLHttpRequest' --data ''

When i look into the logs of the target Alertmanager loadbalancer, i can see that a log like this is created:
xxx.xxx.xxx.xxx - "POST https://unsee.mydomain.com/api/v1/silences" 404 81 "https://unsee.mydomain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"

The domain is wrong! It should be something like this:
xxx.xxx.xxx.xxx - "POST https://alertmanager1.mydomain.com/api/v1/silences" 200 81 "https://unsee.mydomain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"

Any idea how this happens?

Nice find!
https://github.com/cloudflare/unsee/blob/master/proxy.go#L33
I need to look into this part of code and see what's happening there, not sure if I'll have time for this today but I'll get back to you soon. It seems like a simple bug with request headers.

I guess removing the host header from the upstream response could solve the issue: https://github.com/cloudflare/unsee/blob/master/proxy.go#L44

Host header got fixed, now it's time to fix basic auth as well - #256

#256 is merged, can you try the cloudflare/unsee:latest docker image and verify if everything works for you now?

@prymitive i can confirm that it is working for us now 👍


Ip4defrag - set IPv4 Id field to match the value of the packets instead of 0

According to the RFC791:

"the identification field is used to distinguish the fragments of one datagram from those of another"
"To assemble the fragments of an internet datagram, an internet protocol module (for example at a destination host) combines internet datagrams that all have the same value for the four fields: identification, source, destination, and protocol".

It is fine to use the Id of the last packet for the output packet because the Id field is used to identify seen flows (all packets in the same flow have the same Id value).
Added fixed code + test that validate it.

Thanks!


Automate Release Process

Goal
The entire build process (except for the final upload of the build artifact to PyPI) is automated. We do not need to perform any manual tests or checks prior to upload to PyPI.
Summary
We will be making regular releases (bug-fixes, new features, improved documentation) for the foreseeable future. Furthermore, we have decided to push a new release for every bug-fix. We need to make sure that our CI/CD process is sufficiently reliable and completely automated so that minimal time is spent on pushing a new release.
Subtasks

[ ] #744
[ ] #745
[ ] #746
[ ] #747
[ ] #748

I hadn't thought of about half of these!

What triggers this whole process?
Is there a simple way to add a notification step at the end to alert the person with the PyPI keys?

Down the road, it'd be nice to check that the unit tests still functioned without the optional dependencies, too.  When I last checked, the tests passed (by skipping any that used torch) but who knows if that is true anymore?

The current solution for Changelog is to click "create changelog". Is the current changelog sufficient? Then we can remove that step.

What triggers this whole process?

To trigger the release action (logic here):

click on the 'Actions' tab
click on the 'Release' button in the left-hand sidebar
click on the 'Run Workflow' button
Select the 'main' branch and input the Sem Ver that you wish to use
click 'Run Workflow' green button
keep an eye on the progress of the action
if it finishes successfully, it should appear on the repo landing page in the 'Releases' section

Is there a simple way to add a notification step at the end to alert the person with the PyPI keys?

Ping them on Slack. ;) You can also automate this by creating an incoming webhook on Slack and having the action send the status to this URL (I think that Slack provides an action for this in the GitHub Marketplace so all you have to do is store the URL as a repo secret). But probably just manually pinging someone on Slack is easiest.

do not think this is a good idea, we should add tag to each version, as a snapshot for each semver, to allow us track the bug or docs for each release version.
And BTW, if we have tag, we can use auto push to pypi with GitHub tag push event, we can use https://github.com/pypa/gh-action-pypi-publish to auto push package to pypi and that is the best practice for python project
if you guys agree with that, I can change our ci to do that

and if we have a tag in git metadata, the changelog can be auto generate base on the commit, diff from current release tag and previous tag, so I thing https://github.com/SuperDuperDB/superduperdb/issues/747 can be solve, and if we want to show our change log in our document we can use https://sphinx-github-changelog.readthedocs.io/en/latest/ this package to do that, It can reuse the changelog in GitHub, I think it is great which allow thing only in the one place and reused in multiple.
you can see some example in https://air2phin.readthedocs.io/en/latest/ is a repo main

@zhongjiajie Thanks for these suggestions.
I have tried to include some of your suggestions in the PR here: https://github.com/SuperDuperDB/superduperdb/pull/803
We now attach a tag on release. We will open a new PR to add the logic to upload to PyPI automatically.
We have decided not to include the changelog in the documentation for now. But, we may decide to do this in the future. Thanks for letting us know about this feature.
If you are interested, please feel free to add yourself as a reviewer on the PR.

Closing this as all stories are now either finished or in review. Will reopen if any surprises emerge.


[BUG]  generator cards not aligned in center on mobile devices

Description
The generator cards not aligned in center.
I can fix this issue
@Dun-sin can you please assigned it to me.
Screenshots

Thank you for creating this issue. Wait until you are allowed to work on this before you start, and do not forget to include this issue in your pull request. While you wait, don't forget to join the community - [Automated].

Thank you for creating this issue. Wait until you are allowed to work on this before you start, and do not forget to include this issue in your pull request. While you wait, don't forget to join the community - [Automated].

Wow, you just created your first issue on this project, that's so cool. Make sure to wait till you are allowed to work on this before you do and when you are allowed to, don't forget to mention this issue in your pull request. Also join the community - [Automated]

sure go ahead

:tada: This issue has been resolved in version 1.10.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


make[3]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/rte.subdir.mk:35: net] Error 2

centos8.3
LD ixgbe_ethdev.o
AR librte_pmd_ixgbe.a
INSTALL-LIB librte_pmd_ixgbe.a
AR librte_pmd_octeontx2.a
INSTALL-LIB librte_pmd_octeontx2.a
PMDINFO i40e_ethdev.o.pmd.c
CC i40e_ethdev.o.pmd.o
LD i40e_ethdev.o
AR librte_pmd_i40e.a
INSTALL-LIB librte_pmd_i40e.a
make[3]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/rte.subdir.mk:35: net] Error 2
make[3]: *** Waiting for unfinished jobs....
AR librte_pmd_octeontx_crypto.a
INSTALL-LIB librte_pmd_octeontx_crypto.a
make[2]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/rte.sdkbuild.mk:46: drivers] Error 2
make[1]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/rte.sdkroot.mk:99: all] Error 2
make[1]: Leaving directory '/root/ccc/kdns-master/dpdk-19.08'
make: *** [Makefile:51: dpdk] Error 2
[root@ts2280 kdns-master]#

I assume make is run with option -j hence we have 2 issues at same place

INSTALL-LIB librte_pmd_i40e.a
INSTALL-LIB librte_pmd_octeontx_crypto.a

Looks like file or folder permission issue. Can you please add details on the

path it is getting installed?
permission of the folder you are installing?
least can you try running with sudo?

I assume make is run with option -j hence we have 2 issues at same place

INSTALL-LIB librte_pmd_i40e.a
INSTALL-LIB librte_pmd_octeontx_crypto.a

Looks like file or folder permission issue. Can you please add details on the

path it is getting installed?
permission of the folder you are installing?
least can you try running with sudo?

make[1]: Entering directory '/root/ccc/kdns-master/core'
CC dns.o
CC domain_store.o
CC packet.o
CC query.o
CC radtree.o
CC util.o
CC view.o
CC zone.o
AR libkdns.a
INSTALL-LIB libkdns.a
make[1]: Leaving directory '/root/ccc/kdns-master/core'
make[1]: Entering directory '/root/ccc/kdns-master/src'
CC main.o
/root/ccc/kdns-master/src/main.c: In function ‘set_thread_affinity’:
/root/ccc/kdns-master/src/main.c:126:23: warning: comparison is always true due to limited range of data type [-Wtype-limits]
for (cid = 0; cid < RTE_MAX_LCORE; ++cid) {
^
/root/ccc/kdns-master/src/main.c:145:23: warning: comparison is always true due to limited range of data type [-Wtype-limits]
for (cid = 0; cid < RTE_MAX_LCORE; cid++) {
^
/root/ccc/kdns-master/src/main.c: At top level:
cc1: warning: unrecognized command line option ‘-Wno-address-of-packed-member’
CC dns-conf.o
CC parser.o
CC netdev.o
CC forward.o
CC db_update.o
CC webserver.o
CC domain_update.o
CC view_update.o
CC kdns-adap.o
CC tcp_process.o
CC local_udp_process.o
CC process.o
CC hashMap.o
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
cc1: warning: unrecognized command line option ‘-Wno-address-of-packed-member’
make[2]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/internal/rte.compile-pre.mk:116: hashMap.o] Error 1
make[1]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/rte.extapp.mk:15: all] Error 2
make[1]: Leaving directory '/root/ccc/kdns-master/src'
make: *** [Makefile:62: kdns] Error 2

make[1]: Entering directory '/root/ccc/kdns-master/core'
CC dns.o
CC domain_store.o
CC packet.o
CC query.o
CC radtree.o
CC util.o
CC view.o
CC zone.o
AR libkdns.a
INSTALL-LIB libkdns.a
make[1]: Leaving directory '/root/ccc/kdns-master/core'
make[1]: Entering directory '/root/ccc/kdns-master/src'
CC main.o
/root/ccc/kdns-master/src/main.c: In function ‘set_thread_affinity’:
/root/ccc/kdns-master/src/main.c:126:23: warning: comparison is always true due to limited range of data type [-Wtype-limits]
for (cid = 0; cid < RTE_MAX_LCORE; ++cid) {
^
/root/ccc/kdns-master/src/main.c:145:23: warning: comparison is always true due to limited range of data type [-Wtype-limits]
for (cid = 0; cid < RTE_MAX_LCORE; cid++) {
^
/root/ccc/kdns-master/src/main.c: At top level:
cc1: warning: unrecognized command line option ‘-Wno-address-of-packed-member’
CC dns-conf.o
CC parser.o
CC netdev.o
CC forward.o
CC db_update.o
CC webserver.o
CC domain_update.o
CC view_update.o
CC kdns-adap.o
CC tcp_process.o
CC local_udp_process.o
CC process.o
CC hashMap.o
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
cc1: warning: unrecognized command line option ‘-Wno-address-of-packed-member’
make[2]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/internal/rte.compile-pre.mk:116: hashMap.o] Error 1
make[1]: *** [/root/ccc/kdns-master/dpdk-19.08/mk/rte.extapp.mk:15: all] Error 2
make[1]: Leaving directory '/root/ccc/kdns-master/src'
make: *** [Makefile:62: kdns] Error 2

I assume make is run with option -j hence we have 2 issues at same place

INSTALL-LIB librte_pmd_i40e.a
INSTALL-LIB librte_pmd_octeontx_crypto.a

Looks like file or folder permission issue. Can you please add details on the

path it is getting installed?
permission of the folder you are installing?
least can you try running with sudo?

I'm use root

I assume make is run with option -j hence we have 2 issues at same place

INSTALL-LIB librte_pmd_i40e.a
INSTALL-LIB librte_pmd_octeontx_crypto.a

Looks like file or folder permission issue. Can you please add details on the

path it is getting installed?
permission of the folder you are installing?
least can you try running with sudo?

I'm use root

I assume make is run with option -j hence we have 2 issues at same place

INSTALL-LIB librte_pmd_i40e.a
INSTALL-LIB librte_pmd_octeontx_crypto.a

Looks like file or folder permission issue. Can you please add details on the

path it is getting installed?
permission of the folder you are installing?
least can you try running with sudo?

I'm use root

but from your previous reply, you already some insight to the problem
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~

The error you have shared is not coming from DPDK library compilation but from kdns-master/src/hashMap.c linking. Is this a new error?

I assume make is run with option -j hence we have 2 issues at same place

INSTALL-LIB librte_pmd_i40e.a
INSTALL-LIB librte_pmd_octeontx_crypto.a

Looks like file or folder permission issue. Can you please add details on the

path it is getting installed?
permission of the folder you are installing?
least can you try running with sudo?

I'm use root

but from your previous reply, you already some insight to the problem
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~

The error you have shared is not coming from DPDK library compilation but from kdns-master/src/hashMap.c linking. Is this a new error?

I think I am to narrow down to your original problem. The project kdns has Makefile in the parent Folder, where the expected RTE_TARGET is x86_64-$(machine)-linuxapp-gcc. From your logs the RTE_TARGET is set to arm64-armv8a-linuxapp-gcc.
To cross-check if this is the problem, I tried the following

git clone the project
untar DPDK-19.08
cd DPDK folder
export RTE_TARGET=x86_64-native-linuxapp-gcc
export RTE_SDK=$PWD
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
cd x86_64-native-linuxapp-gcc/
make -j (dpdk is built successfully)
cd ../../
make

I get the dependency errors as
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libmicrohttpd/src/microhttpd/.libs/libmicrohttpd.a: No such file or directory
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libjansson/src/.libs/libjansson.a: No such file or directory

and not
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int

I think I am to narrow down to your original problem. The project kdns has Makefile in the parent Folder, where the expected RTE_TARGET is x86_64-$(machine)-linuxapp-gcc. From your logs the RTE_TARGET is set to arm64-armv8a-linuxapp-gcc.
To cross-check if this is the problem, I tried the following

git clone the project
untar DPDK-19.08
cd DPDK folder
export RTE_TARGET=x86_64-native-linuxapp-gcc
export RTE_SDK=$PWD
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
cd x86_64-native-linuxapp-gcc/
make -j (dpdk is built successfully)
cd ../../
make

I get the dependency errors as
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libmicrohttpd/src/microhttpd/.libs/libmicrohttpd.a: No such file or directory
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libjansson/src/.libs/libjansson.a: No such file or directory

and not
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int

I think I am to narrow down to your original problem. The project kdns has Makefile in the parent Folder, where the expected RTE_TARGET is x86_64-$(machine)-linuxapp-gcc. From your logs the RTE_TARGET is set to arm64-armv8a-linuxapp-gcc.
To cross-check if this is the problem, I tried the following

git clone the project
untar DPDK-19.08
cd DPDK folder
export RTE_TARGET=x86_64-native-linuxapp-gcc
export RTE_SDK=$PWD
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
cd x86_64-native-linuxapp-gcc/
make -j (dpdk is built successfully)
cd ../../
make

I get the dependency errors as
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libmicrohttpd/src/microhttpd/.libs/libmicrohttpd.a: No such file or directory
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libjansson/src/.libs/libjansson.a: No such file or directory

and not
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int

Solution you can try is to edit Makefile for your arm target.

In the x86 platform can be compiled normally through, I this error occurred under the arm64 platform. Then I commented on a line that I could compile through ：
`/**

@warning
@b EXPERIMENTAL: this API may change without prior notice.

try to take a read lock.

@param rwl
A pointer to a rwlock structure.
@return

zero if the lock is successfully taken

-EBUSY if lock could not be acquired for reading because a

writer holds the lock

*/
//__rte_experimental     //<-- here
static inline int
rte_rwlock_read_trylock(rte_rwlock_t *rwl)
{
`

I think I am to narrow down to your original problem. The project kdns has Makefile in the parent Folder, where the expected RTE_TARGET is x86_64-$(machine)-linuxapp-gcc. From your logs the RTE_TARGET is set to arm64-armv8a-linuxapp-gcc.
To cross-check if this is the problem, I tried the following

git clone the project
untar DPDK-19.08
cd DPDK folder
export RTE_TARGET=x86_64-native-linuxapp-gcc
export RTE_SDK=$PWD
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
cd x86_64-native-linuxapp-gcc/
make -j (dpdk is built successfully)
cd ../../
make

I get the dependency errors as
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libmicrohttpd/src/microhttpd/.libs/libmicrohttpd.a: No such file or directory
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libjansson/src/.libs/libjansson.a: No such file or directory

and not
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int

Solution you can try is to edit Makefile for your arm target.

you can try :
make deps && make kdns

arm64-armv8a-linuxapp-gcc

Looks like there is gap here, as per logs you have shared the RTE_TARGET set on your build environment is arm64-armv8a-linuxapp-gcc. If you are intending to run on x86 or x86_64 you should not use arm64-armv8a-linuxapp-gcc.

I think I am to narrow down to your original problem. The project kdns has Makefile in the parent Folder, where the expected RTE_TARGET is x86_64-$(machine)-linuxapp-gcc. From your logs the RTE_TARGET is set to arm64-armv8a-linuxapp-gcc.
To cross-check if this is the problem, I tried the following

git clone the project
untar DPDK-19.08
cd DPDK folder
export RTE_TARGET=x86_64-native-linuxapp-gcc
export RTE_SDK=$PWD
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
cd x86_64-native-linuxapp-gcc/
make -j (dpdk is built successfully)
cd ../../
make

I get the dependency errors as
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libmicrohttpd/src/microhttpd/.libs/libmicrohttpd.a: No such file or directory
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libjansson/src/.libs/libjansson.a: No such file or directory

and not
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int

Solution you can try is to edit Makefile for your arm target.

you can try :
make deps && make kdns

Thanks for your suggestion, as mentioned above I have no intention to run Kdns and I am not owner or contributor for the project. I have replied and tried the steps since it was more of DPDK related setting issue.
For better help on KDNS reach out to author or contributors. I am done here.

on arm64 ,I fixed a bug
//set thread's affinity to cpus that are not used by dpdk
static int set_thread_affinity(void) {
int s;
uint8_t cid;
pthread_t tid;
cpu_set_t cpuset;
unsigned long long cpumask = 0;
tid = pthread_self();
CPU_ZERO(&cpuset);

++    for (cid = 0; cid < RTE_MAX_LCORE-1; ++cid) {
++    for (cid = 0; cid < RTE_MAX_LCORE; ++cid) {
        if (!rte_lcore_is_enabled(cid)) {
        CPU_SET(cid, &cpuset);
    }
}

s = pthread_setaffinity_np(tid, sizeof(cpu_set_t), &cpuset);
if (s != 0) {
    log_msg(LOG_ERR, "fail to set thread affinty, errno=%d, errinfo=%s\n", errno, strerror(errno));
    return -1;
}

CPU_ZERO(&cpuset);
s = pthread_getaffinity_np(tid, sizeof(cpu_set_t), &cpuset);
if (s != 0) {
    log_msg(LOG_ERR, "fail to get thread affinity, errno=%d, errinfo=%s\n", errno, strerror(errno));
    return -2;
}

++  for (cid = 0; cid < RTE_MAX_LCORE-1; cid++) {
--  for (cid = 0; cid < RTE_MAX_LCORE; cid++) {
if (CPU_ISSET(cid, &cpuset)) {
cpumask |= (1LL << cid);
}
}
return 0;

}

I think I am to narrow down to your original problem. The project kdns has Makefile in the parent Folder, where the expected RTE_TARGET is x86_64-$(machine)-linuxapp-gcc. From your logs the RTE_TARGET is set to arm64-armv8a-linuxapp-gcc.
To cross-check if this is the problem, I tried the following

git clone the project
untar DPDK-19.08
cd DPDK folder
export RTE_TARGET=x86_64-native-linuxapp-gcc
export RTE_SDK=$PWD
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
cd x86_64-native-linuxapp-gcc/
make -j (dpdk is built successfully)
cd ../../
make

I get the dependency errors as
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libmicrohttpd/src/microhttpd/.libs/libmicrohttpd.a: No such file or directory
/usr/bin/ld: cannot find /tmp/kdns/src/../deps/libjansson/src/.libs/libjansson.a: No such file or directory

and not
In file included from /root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/rte_rwlock.h:12:0,
from /root/ccc/kdns-master/src/hashMap.c:1:
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:92:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int
^~~~~~
/root/ccc/kdns-master/dpdk-19.08/arm64-armv8a-linuxapp-gcc/include/generic/rte_rwlock.h:136:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘static’
static inline int

Solution you can try is to edit Makefile for your arm target.

you can try :
make deps && make kdns

Thanks for your suggestion, as mentioned above I have no intention to run Kdns and I am not owner or contributor for the project. I have replied and tried the steps since it was more of DPDK related setting issue.
For better help on KDNS reach out to author or contributors. I am done here.

thanks so much
I've modified the makefile file, and I've found some other issues that have been fixed and are now up and running on the arm platform.
#/* Copyright (c) 2018 The TIGLabs Authors */
ifdef V
Q =
else
Q = @
endif
ifeq ($(machine),)
machine = armv8a
endif
ifeq ($(JOBS),)
JOBS := $(shell grep -c ^processor /proc/cpuinfo 2>/dev/null)
ifeq ($(JOBS),)
JOBS := 1
endif
endif
#RTE_SDK_TAR = $(CURDIR)/dpdk-19.08.tar.xz
#RTE_SDK = $(CURDIR)/dpdk-19.08
RTE_SDK_TAR = $(CURDIR)/dpdk-19.11.6.tar.xz
RTE_SDK = $(CURDIR)/dpdk-stable-19.11.6
export RTE_SDK
Default target, can be overriden by command line or environment
RTE_TARGET ?= arm64-$(machine)-linuxapp-gcc
export RTE_TARGET
bindir =  $(CURDIR)/bin


Alum access to read the lunch menu

would be nice

Lunches aren't really a thing anymore, and I'm not even sure if I'd be allowed to do this.


'babelHelpers.asyncToGenerator is not a function' error only on iOS

Is this a bug report?
Yes
Have you read the Contributing Guidelines?
No
Environment
Environment:
OS:  macOS Sierra 10.12.1
Node:  7.9.0
Yarn:  0.20.3
npm:  4.2.0
Watchman:  4.9.0
Xcode:  Xcode 8.3.3 Build version 8E3004b
Android Studio:  2.3 AI-162.4069837
Packages: (wanted => installed)
react: 16.0.0-alpha.12 => 16.0.0-alpha.12
react-native: 0.48.3 => 0.48.3
Target Platform: iOS (10.3) Android
Expected Behavior
App to load as it does on android .
Actual Behavior
Error: babelHelpers.asyncToGenerator is not a function

Duplicate of #4844

I got same error in react-native 0.50.4.
I think that babelHelpers.asyncToGenerator is missing in Libraries/polyfills/babelHelpers.js, right ?
It works fine after add babelHelpers.asyncToGenerator to Libraries/polyfills/babelHelpers.js.
(It is partial of require('babel-core').buildExternalHelpers('asyncToGenerator'.split(' ')).)
babelHelpers.asyncToGenerator = function (fn) {
  return function () {
    var gen = fn.apply(this, arguments);
    return new Promise(function (resolve, reject) {
      function step(key, arg) {
        try {
          var info = gen[key](arg);
          var value = info.value;
        } catch (error) {
          reject(error);
          return;
        }

        if (info.done) {
          resolve(value);
        } else {
          return Promise.resolve(value).then(function (value) {
            step("next", value);
          }, function (err) {
            step("throw", err);
          });
        }
      }

      return step("next");
    });
  };
};

It is bug ? I made a mistake ?

Also having issues with this. Using babel-preset-react-native makes no difference, unfortunately.

package.json
{
        ...
	"dependencies": {
		"color": "^2.0.1",
		"expo": "^25.0.0",
		"moment": "^2.20.1",
		"prop-types": "^15.6.0",
		"ramda": "^0.25.0",
		"ramda-extension": "^0.2.1",
		"react": "^16.2.0",
		"react-native": "https://github.com/expo/react-native/archive/sdk-25.0.0.tar.gz",
		"react-native-actionsheet": "^2.3.0",
		"react-native-communications": "^2.2.1",
		"react-native-datepicker": "^1.6.0",
		"react-native-elements": "^0.18.5",
		"react-native-iphone-x-helper": "^1.0.1",
		"react-native-pathjs-charts": "0.0.33",
		"react-native-swipeable": "^0.6.0",
		"react-native-swiper": "^1.5.13",
		"react-navigation": "^1.0.0-beta.27",
		"react-redux": "^5.0.6",
		"react-search-input": "^0.11.3",
		"redux": "^3.7.2",
		"redux-api-middleware": "^2.2.0",
		"redux-logger": "^3.0.6",
		"redux-persist": "^5.4.0",
		"redux-thunk": "^2.2.0",
		"reselect": "^3.0.1"
	},
	"devDependencies": {
		"babel-eslint": "^8.2.1",
		"babel-plugin-module-alias": "^1.6.0",
		"babel-plugin-transform-react-jsx-source": "^6.22.0",
		"babel-plugin-module-resolver": "^3.0.0",
		"babel-plugin-transform-function-bind": "^6.22.0",
		"babel-preset-env": "^1.6.1",
		"babel-preset-react-native": "^4.0.0",
		"eslint": "^4.15.0",
		"eslint-config-airbnb": "^16.1.0",
		"eslint-import-resolver-babel-module": "^4.0.0",
		"eslint-loader": "^1.9.0",
		"eslint-plugin-import": "^2.8.0",
		"eslint-plugin-jsx-a11y": "^6.0.3",
		"eslint-plugin-react": "^7.5.1",
		"eslint-plugin-react-native": "^3.2.0",
		"jest-expo": "^25.0.0",
		"quokka-signet-explorer": "^1.3.0",
		"remote-redux-devtools": "^0.5.12"
	}
}

After doing the dependency + cache nuking a handful of times it started to work again.

removed node_modules
cleared $TMPDIR files/directories
cleared yarn cache
reinstall dependencies
opened XDE
turned on development mode and restarted project
turned off development mode and restart project


New About page

Problem
The current site displays content related to various aspects of the project on multiple pages in the homepage header. Due to this, the navigation component appears visually cluttered and lacks space for more items. Additionally, there are a few tickets in the repository requesting additional content to explain how both existing and upcoming features work.
It is necessary to revisit how we present Openverse to both current and prospective users and devise a solution that helps visitors easily locate all relevant information.
Related issues
The original idea comes from this comment in #3564, but I went back to all tickets tagged with design: needed requesting for adding content, and the following ones relate to this topic in one way or another.

#4303, as part of #431 project
#459
#611
#2595

Proposal
Given that all the information relates to Openverse's essence, the About page meets the goal perfectly. But before showing the proposals, I put down this brief to align our expectations over this idea:
Main goal:

Introduce Openverse as a tool that surfaces openly-licensed content from the web.

Secondary goals

Move most content from current header pages into a single one under a storytelling that goes from broad concept into the specs.
Create a more appealing page by pushing the brand further

Users

New visitors who know nothing about Openverse
Current and/or expert users looking for how Openverse works

Offset box grid (i1.2)
https://github.com/WordPress/openverse/assets/895819/380fdea1-73a2-40ff-ae19-85616165065e
https://github.com/WordPress/openverse/assets/895819/91de64b1-21f0-43d9-a2af-92368a09bfc4
This version introduces Openverse by playing with the spacings of the "All content" results grid to reach a more appealing version. The result is real media content from the "jazz" search and each element links to their respective details page. The dark section below describes more the concept showed above and includes some source logos to reinforce the search-engine core value of the project.
Funky layout (i1.5)
https://github.com/WordPress/openverse/assets/895819/65ae43ec-de44-4a7a-b980-69e5362ba809
https://github.com/WordPress/openverse/assets/895819/d3a2cff2-c6bb-454a-966f-7ed4766700a6
This version distributes image and audio track items in a device's full-height area with a jazzy layout and a big heading at the center. The dark section below is the same as in the previous version.

Next steps
If you notice, there are a few style tweaks on the homepage as well to connect the visual language used. In that vein, this proposal also relates to a segment of #4157's designs. Meanwhile, I will ping comms folk asking for help writing the page. I envision big collaboration between folks as the tickets related mentioned above involve expert knowledge and technical terms that will require careful revision.
But most importantly, I would like you thoughts about the versions and which one feels better. Check the links below to interact with the prototypes, although some elements are not fully accurate with the interaction, you will get the main idea.
Mockups

Figma: New About page
All prototypes. Click on the top left sidebar action to see them all.

Lovely work. Lots to like.
I'm personally most drawn to this one:

Random is hard to get right. As a fashionista once said, and I'm probably paraphrasing: "it takes 45m to look like you just got out of bed". Random and disorderly like this is the same, true random rarely works. There needs to be some pattern in the chaos for you to recognize and latch on to. To be fair, there is such a pattern here, it's carefully scattered between light and dark around the center, and generally legible. If this is the one your heart beats for, can definitely work. But there's something very curated about the other gridded one, that looks great.

I also prefer the grid design, actually a lot more. Not just aesthetically, but also technically. It's easier to imagine how it would actually scale in and out with different breakpoints. Implementation in general would be more straightforward than the other design, and considering this in balance with other work Openverse is doing, that strikes me as an important consideration. I don't think we'd need any new positioning or organisation tools to implement it, whereas the jazzy layout would need entirely new approaches, ones that are sure to be complex (in particular full of edge cases).
For example, I'm a bit concerned about how the audio controls in particular would behave as the page scales within a breakpoint. We'd need to be at least be careful that the "pause" button was never hidden for any of the playable audio. Keyboard navigation for sighted users also seems hectic when I try to imagine it, unless we would skip those interactive elements for keyboard navigation. They are essentially presentational, but the demo highlights the interaction, and indeed it would be strangely frozen not to make them interactive if they were there. Speaking more generally (rather than about the theoretical technical implementation), the cut-off titles of the audio also give me a sense of disconnected context, like I was meant to be able to scroll up further, but there wasn't any further to scroll, or worse, like I'm drowning, caught in an up-close and claustrophobic view of the centre of some unknown whole. The edges of the images skew and bend due to some illusion. It's actually a fascinating experience, but maybe too dizzying or intense for the about page of what is just a search engine :sweat_smile:. Anyway, the gridded option is calmer and less challenging in this sense, at least for me.
I'm unclear whether you're also looking for feedback on some of the general language you've used for the leader text and the goals, and I understand there'll be collaboration for writing the full text of the page. However, given the prominence of this on the design, I want to share that I don't think that the verb "surfacing" is a clear or easy to follow one. For one, it comes across business/corporate-y (maybe ironic given jazz as the design inspiration). "Search the web's open content" is proximate to other tools users will be familiar with (Google Search, "search engine", etc), matches ways in which we already describe Openverse, and also acts as a call to action. "Surfacing" is about what Openverse does, "Search" is about what the user can do with Openverse (at least, that's how I interpret the difference it). Surfacing is also, incidentally, passive voice, which comes across hollow when it's the text leading the page.

The box grid is the better option for practical concerns (like accessibility, responsiveness and technical implementation), although I love the look of the funky one more. I agree with the points raised by @sarayourfriend in the first two paragraphs.
As for language, I want to dissent. As the "About" page for Openverse, it should be about what Openverse does and so "surfacing" feels right to me. But I won't harp on it because a skilled copywriter can do a much better job of finding the right text to show here.
I like the use of B/W images on this page very much, so much so that I would recommend using B/W images exclusively and no color images. They look striking and go very well with our brand colors.
We could include a couple of audio objects in the grid layout as well, similar to the funky one, provided we assuage @sarayourfriend's concerns by ensuring that important parts of the audio component do not overflow off-screen.

I agree with a lot of the feedback so far! I would love to see if adding the audio players and the idea of having the elements extend out of the viewport could be added to the grid based design.
I really like the second block as well, which showcases some logos of the different sources in Openverse.
I do think the existing About page needs a major content overhaul. I would also like to see more "content block" styles to use further down the page. Even if they were initially designed with placeholder copy, I am confident we could use them with new copy once we have it with minimal modification.
I love where this is heading!

Thanks for all the feedback shared.
The jazzy version was intentionally shared to spark ideas about how folks felt about it and what opportunities we had implementing it while making it accessible. I also considered treating it just as a decorative image, without interaction. But if the grid version feels more coherent with the brand, then let's keep pushing that one.
I will iterate on a new version with the following in mind:

Add audio content to the grid
Try black and white images to keep the yellow + black duotone of the brand
I created this document and will ping Lauren asking for help. Comments on the copy will be included.
Consider other content blocks to style the biggest body content area.

Given that the next iteration will be based on the box grid, I wonder what options we have for the breakpoints behavior. I will propose something but it's open to new ideas for playing with the grid's off spaces and reach an appealing hero section for every viewport.
Regarding the page content, I think we've used the word "surfacing" to make web content visible through the search engine. And as @dhruvkb pointed out, it's in the context of describing Openverse as a product. I might be wrong about its previous uses, but in any case, especially if the word unfits, it would be good to have a set of words and sentences we can refer to here and in all Openverse comms to have a consistent voice tone.
Thanks again for the all your thoughts.

+1 to the grid layout, though the jazzy layout is very fun! The marque'd CC logos and rotating provider logos do give me an idea though: I'm not sure if this was intentional, but each of those could be a clickable element which then directs to a search using that as a filter! For instance, the CC license rotating items could link to a search filtered by that license, and the "SMK logo" in the black and white section below would link to an image search filtered down to SMK results. Just a thought 😄

@AetherUnbound We don't support query-less searches on the frontend at the moment, and don't have UI for it either. For the providers we can use the collection pages, but for licenses we don't have that and would need to create it. Just giving context for the scope of that suggestion, basically would be an entire new page rather than just a link to an existing page.
Regarding "surfacing", I don't see a single instance of it in the Openverse codebase to describe Openverse (it is used in one IP, but that's not project copy), so I do think it is new, and it is a significantly different way/mode of describing Openverse. I look forward to discussing it, and agree that more comprehensive guidance on the specific words used to describe Openverse in our copy would make things like this easier moving forward.

@AetherUnbound We don't support query-less searches on the frontend at the moment, and don't have UI for it either. For the providers we can use the collection pages, but for licenses we don't have that and would need to create it. Just giving context for the scope of that suggestion, basically would be an entire new page rather than just a link to an existing page.

You're totally right, that slipped my mind. The collection pages were what I was thinking of, though that does not extend to licenses in the way I described.

@AetherUnbound We don't support query-less searches on the frontend at the moment, and don't have UI for it either. For the providers we can use the collection pages, but for licenses we don't have that and would need to create it. Just giving context for the scope of that suggestion, basically would be an entire new page rather than just a link to an existing page.

To narrow the scope of this, we could use a random word of the top searches (based on Plausible data?), with that license selected. For instance, if you click on "CC0", you would go to https://openverse.org/search/?q=sky&license=cc0.

I don't want to neutralize the creative energy you are putting on this, but since we don't have ways to show results filtered by a CC license without a search term, it sounds easier to treat it as a decorative element.

I agree

Thanks for the ping @fcoveram. I'd love to help out with revisiting/refining/refreshing the Openverse About page.
For what it's worth, I'd also throw my vote behind the grid version.

I would also like to see more "content block" styles to use further down the page.

I don't know if this aligns with what you meant, @zackkrida, but I wonder what's limiting the design from including other visuals interspersed with the content. Whether it's images or audio, it feels like a golden opportunity to tell AND show what Openverse has available (beyond a line of logos) and what it can do.

Regarding "surfacing", I don't see a single instance of it in the Openverse codebase to describe Openverse (it is used in one IP, but that's not project copy), so I do think it is new, and it is a significantly different way/mode of describing Openverse. I look forward to discussing it, and agree that more comprehensive guidance on the specific words used to describe Openverse in our copy would make things like this easier moving forward.

I think @sarayourfriend brings up a good point about what language to use and what might sound like Openverse. Does Openverse have a particular way of speaking? Is it meant to be different from WordPress's brand voice and tone? If it's of interest, maybe it's worth exploring that identity with this About page refresh along with the design.

… I wonder what's limiting the design from including other visuals interspersed with the content. Whether it's images or audio, it feels like a golden opportunity to tell AND show what Openverse has available (beyond a line of logos) and what it can do.

I don't envision significant limitations beyond adding component interactions and developing new elements that would require content integrations. In any case, we can iterate @thetinyl in the document and drop ideas as text or images to discuss them before any implementation that could push the project scope further. I agree with the opportunity to enhance the storytelling and convey something meaningful and appealing.

Does Openverse have a particular way of speaking? Is it meant to be different from WordPress's brand voice and tone? If it's of interest, maybe it's worth exploring that identity with this About page refresh along with the design.

I think we haven't talked about this before. And I personally don't see Openverse with a different brand voice and tone. To keep Openverse under the WordPress project umbrella, I would apply the guidelines already made and see how we feel about it. So far, all comms pieces feel consistent and robust with WordPress.

And I personally don't see Openverse with a different brand voice and tone. To keep Openverse under the WordPress project umbrella, I would apply the guidelines already made and see how we feel about it. So far, all comms pieces feel consistent and robust with WordPress.

This is what I've noticed until now, and didn't want to assume.

I chatted with @thetinyl internally to coordinate our times, and we can start crafting the page content the week of June 24th.
However, the most significant dev effort seems to be in the hero section where media components are. For the rest of the page, the sections intend to work as blocks arrangement that don't reshape the TOC area. Therefore, I can split the design file into two sections to begin implementing this first part separately without depending on having the whole content.
How does that sound?

I'd like clarification on how this would be prioritised, and what the expectations for implementing this are. @zackkrida can you help out with that? As it stands, I don't know where this fits in with the rest of our work, of which there is, to say it plainly, quite a lot.
I understand the motivation behind the page, but how is that balanced against the rest of the work we're doing? What do we hope to get out of this compared to the other things we're planning to work on? Etc. Would be great to have input from @zackkrida to clarify that, in collaboration with you @fcoveram, and in light of all the other things we've got going on.

In case we need to speed up the implementation, or if this design expands the times and scope initially assessed significantly, we could go with adding content needed for #431 and #2595 in the existing pages. Or include a new page is that's an unavoidable case.

@sarayourfriend and @fcoveram my feeling is that this design is great to have prepared, but that we should defer implementing these changes (and more) until next year.
There is value in having design explorations ready now. The page will need new copy written by our team. I believe that copy is currently blocked by:

A refinement of the goals, audience, and principles governing Openverse
Having clear "key performance indicators" (KPIs) for the project

@fcoveram I think you could continue to explore with another 1-2 "generic" content sections to expand the page, and then we could mark this as blocked until we have concrete plans to implement it.

@fcoveram apologies, by "the project" I meant Openverse itself, not this specific "about page" project.
The main reason to block this is the team's work load and the fact that the page needs new copy.
I thought creating some generic sections could be a nice way to continue design explorations despite these blockers.
I am not exactly sure what you mean by "the page layout does not change significantly". Could you clarify? It seems okay for the about page to have a strong presentation that uses different text styles and layouts from the main search experience.

The main reason to block this is the team's work load and the fact that the page needs new copy.

That makes sense. I still believe that @thetinyl could write a draft document that we only need to revisit in terms of technical accuracy rather than voice and tone. Ticket #2595 could require more our attention as it involves new content.

I am not exactly sure what you mean by "the page layout does not change significantly". Could you clarify? It seems okay for the about page to have a strong presentation that uses different text styles and layouts from the main search experience.

Sorry for not being clear. I meant that the main content area will remain simple and using existing styles. Here is a quick mockups with notes of how the rest of sections intend to be.

Because of the same workload point is that I worked on a proposal that only introduces a new section at the beginning of the page while keeping the remaining areas simple without requiring extra code work.

I don't want to sound rigid about my position, but I would like to address this as the number of pages in header and the content clean up are the core goals of this proposal.

I'm happy to support with copy in whatever capacity makes sense for the team/process. Whether that's sooner or later. (Just wanted to throw that out there.)

@fcoveram thanks for clarifying your goals, that helps me understand a lot better where you're coming from. My point of view has been focused on the "Introduce Openverse as a tool that surfaces openly-licensed content from the web." goal and not the "number of pages in header and the content clean up" goals.
To be honest, I was so enamored with the "Hero" designs that I overlooked your goal to consolidate all of the existing pages into the "About" page. I actually don't think that is the best idea. I would rather the "About" page work like a traditional home page does, with more succinct copy explaining different facets of the project, and not have long "walls of text" like our more content-heavy pages.
Perhaps something like the sub pages on Wordpress.org's About page could work: https://wordpress.org/about/ but that would also require some significant work.
If the highest-priority goal is reducing the number of pages in the header, could we instead consider implementing a dropdown menu for many of the pages? Or another approach? I think we might want to separate the goals of creating a beautiful, narrative "About" page and cleaning up the Openverse nav. Another example: We could probably move "search help" from the site header and move it into the "recent searches" dropdown as a text link.
Also keep in mind, users spend the most time on the search view, where the header is moved to the footer of the site. The number of pages is less of a concern there.
what do you think @WordPress/openverse?

A few thoughts from me:

It'd be nice if content heavy pages weren't so tedious to code. Writing them in Markdown and rendering them into our Vue pages, for example, would make things a lot easier, though potentially more difficult for translation (would need to confer with GlotPress folks on that one). The existing content heavy pages are extremely tedious (especially in translation) and I'd be reluctant to add more of them without taking the oppotunity to improve things. That being said, this consolidates them, so it's a bit different :shrug:.
I'm indifferent to the content of the About page being short or long. I don't see the difference, especially if it's well organised (the table of contents sidebar makes sense to me, and will be familar to users of other websites with content heavy pages). Again, however, it's a lot of new components and currently manual wiring of things. Fine by me, but we'd need to intentionally push off something else to do this instead.
Dropdown in the homepage nav might be fine. Moving links to context sensitive places (like you suggested for the search help, Zack) makes a lot of sense to me. We could move get involved, feedback, terms, and privacy to the footer of the homepage, next to the WP foundation mark. That makes contextual sense to me, considering they have to do with meta information about the project as a whole rather than search. We could move the sources link out of the nav into the hero text of the homepage as well ("An extensive library of free stock photos, images, and audio, available for free use, thanks to the large community of sources hosting openly licensed media", or something like that) Licenses could also be justifiably relocated from the homepage header in a similar manner. If we don't move those two, that leaves About, Licenses, Sources, and API in the homepage header. Excepting the dropdown, these are all relatively easy changes to make to improve the number of links in the homepage header, if we want to target that problem as a specific subset of our overall page content issues.

In general I like these changes to the about page and the hero and stuff, I'm not opposed to any of them, and I think they could be fine to prioritise. Especially if coupled with taking a step back after the Nuxt 3 migration and making some key architectural decisions about how we want to move forward with the structure of the frontend code (especially the layout methodology), it's a good prompt to do so. We certainly have ample opportunity to improve the technical aspects of the frontend, and content management is no small part of that.
That being said: I wonder if headless WP is an option here, especially if it would in any way make translation and authorship easier for large chunks of text? If we use WP as the CMS for these pages, and Nuxt just SSR renders the layout, TOC, etc, it might make some things better, and would demonstrate a way in which WP can be used to augment other types of pages. Kind of ties in nice with WP foundation concerns and contemporary discussions about using WP as a content backend. (Not to complicate things further, lol!... but if we're dreaming a bit... might as well dream?).

To @zackkrida's points

I would rather the "About" page work like a traditional home page does, with more succinct copy explaining different facets of the project, and not have long "walls of text" like our more content-heavy pages.
Perhaps something like the sub pages on Wordpress.org's About page could work: https://wordpress.org/about/ but that would also require some significant work.

This is also a good idea. It still needs a page introducing the topics with links to the internal pages, but worth exploring this suggestion.

If the highest-priority goal is reducing the number of pages in the header, could we instead consider implementing a dropdown menu for many of the pages?

Reducing the number of links comes from the potential path of adding more due to the tickets mentioned in the first comment. But the current header handles very well the shift from showing all items to the burger menu.

I think we might want to separate the goals of creating a beautiful, narrative "About" page and cleaning up the Openverse nav.

You’re right about this. The design shared approached the solution progressively. So, for this first release, the page aims to gather everything in a single place, while in future steps, we can push the page further and add more enriched content.

We could probably move "search help" from the site header and move it into the "recent searches" dropdown as a text link.

+100 to this. It’s in line with some comments shared before about showing info in context, but it suits better in a new ticket.
To @sarayourfriend's points
Your ideas about creating and editing these pages less tedious sound great. At this point, I don’t know exactly how to continue with the goal of making the header cleaner, either with @zackkrida's idea of something like the WordPress' About page or the design introduced here.
I also agree with you on not having a preference for long or short content, but the current set of links can definitely be introduced more clearly.

…We could move get involved, feedback, terms, and privacy to the footer of the homepage, next to the WP foundation mark. That makes contextual sense to me, considering they have to do with meta information about the project as a whole rather than search…

This could work, and adapting header and footer components to this doesn’t sound like a big task. It’s very straightforward.
I would not move elements out of the header and footer as having them handy from any page, even during the search flow, adds tremendous value. But your suggestions of linking to specific pages from other parts of the UI make plenty of sense to me.

I don’t know exactly how to continue with this, but the two following ideas sound good to me:

Update the About to host the other internal pages. Similar to WordPress’ About page structure.
Move certain links from header to footer and link to these from other parts of the UI.

I’m drawn to take the first path and implement its simplest version within the available time the team has. This solution can grow progressively and become having one or several pages with rich content.
Managing this content definitely requires deep thought, so I would leave in your hands what’s best for now.

@fcoveram your synthesis of the two primary ideas for moving forward sounds great to me! I would be fully on-board with those approaches, given the context Zack and Sara have supplied.

I'm fine with both, and I think we can do the full about page stuff at whatever pace we want, so I'm not sure we need to cut the scope of it. To me the question is: can we break it up into smaller milestones, keeping in mind there are other things that are of extremely high value (search relevancy) that we simply must spend a considerable amount of time looking at as a team. Changes to the header/footer link locations is an easy milestone to achieve, if that will improve things. Merging certain pages into a single About page with the existing layout tools we have (potentially with a minor addition of a ToC) is also an easily achievable milestone. Adding the about page hero could be the final milestone in this series of milestones, perhaps with changes and advancements to how we manage and render static textual content in general. To formalise my suggestion into an ordered list:

Move header links around to either include them in the footer, or make them contextual with existing copy on our home and search pages.
Merge textual content pages into a single about page, with the addition of a simple table of contents widget.
Develop the new about page hero, and potentially consider improvements/advancements to our management and rendering of static textual content.

Breaking it up that way (if that indeed achieves the overall end goal this conversation is getting at) makes it feel less like a potentially huge and gnarly project that would indeed take a significant portion of our attention, to a series of smaller achievements that could be done "on the side" to other things. It gives small, meaningful milestones. Especially because that approach pushes off the most technically complex part of this to the end, when we'll have a clearer view of just how many maintainers and how much of their time will need to be dedicated to the vast and technically complex work of search relevancy (and its various support projects).

I think we can do the full about page stuff at whatever pace we want, so I'm not sure we need to cut the scope of it

Agree

…can we break it up into smaller milestones, keeping in mind there are other things that are of extremely high value (search relevancy) that we simply must spend a considerable amount of time looking at as a team.

Absolutely. And I also agree with prioritizing other more relevant work.

Milestones
Based on @sarayourfriend's suggestions, here is how the iA could evolve.
Step 1. Links in header and footer
In this first milestone, the links are rearranged across the header and footer.

For the above, here are the mockups of the "internal" variants of header and footer.

No changes are needed for the "content" variants of both components. They can continue working as they are.
Step 2. Pages inside the About page
For the second milestone, the About page has minor style tweaks to index links to the pages noted in the diagram below.
These second-level pages would remain as they are in terms of content and style, and slugs could be set, for example, for the "Sources" page, like openverse.org/about/sources.

Step 3. New about page
Finally, the third and final stage introduces the new About page (based on the Offset box grid idea), where all content displayed in the previous internal pages is part of the "About" one.

I have the feeling we can skip step one and go immediately with the second one by applying minor style tweaks. I don't envision the need for a TOC as the page is straightforward and mainly shows links to child pages.
What do you think of this?

Thanks so much for outlining this @fcoveram! I love the plan, but can't make any statement on starting with phase 1 or 2 based on my level of familiarity with the frontend. Either way, I think it's a great evolution.

I have a suggestion for step 2: rather than having nested routes, we could have hash routes for sections. So, sources would be linked to the anchor on openverse.org/about page: openverse.org/about#sources. This way, we won't have to change the URLs and add new redirects, which I think is bad for SEO.

This is what I was thinking for the step 3, and also for the redesigned page.
If I understand correctly, do you mean putting all content on a single page and linking each section through anchors? If so, that sounds good.
I envision each page title in the current structure will live as an h2 element in the About page. Are you also considering a TOC to jump between sections or just having all content in a single place with the same current layout? Both work for me since this is an iterative progress.

The diagram in Step 3 lists "Get involved" twice: both in the footer and in the About page. Should it only be in the footer?

The "get involved" inside the About page is more like a link to reinforce the contribution message. It doesn't need to be the same copy, but it does link to the same place.
Perhaps I didn't explain it correctly, but the yellow boxes in step 3's diagram are pages in both header and footer. Just as it works now when links are visible in the header, they're not in the footer, and vice versa. It shifts depending on which page you are on.

Do I understand it correctly that the About page is mostly information for people using Openverse's frontend and the footer links are legalese and links for people who want to "extend" or "contribute to" Openverse (so, Get involved and API are here)?

Yes. And I would add that the About page is for "understanding and using" Openverse.


Defined ANDROID_API to fix build on Android

In libssh, strerror_r was assumed to return char* when glibc is used, and assume it returns int instead. On Android this was true until after API 23, it switched to use GNU flavor, causing the library failed to build.
This PR specifies 21 as Android API target, fixes the build issue.

Thanks!
Is it possible that someone will need to specify a different value than 21? I'm not really familiar with android!

Hi, I think it's possible, let me try to figure out if the Rust compiler gives the information :)

OK I have done some quick researches!
According to NDK/API Update Policy, Rust supports up to latest LTS NDK, and according to NDK Revision History, they dropped support of Android 4.4, which means the minimum supported API version is 21. So we can safely assume the the most developer only targets API 21+.

Thank you!


ADD THE MORSE CODE MAP AND DECODER

🔤 Feature Implementation: Morse Code Decoder Methods
This pull request contains the implementation of Morse code decoding methods. The implemented methods translate Morse code into string representations of characters, words, and entire messages.
Implemented Morse Code Decoding Methods:

decode_char method: Decodes a Morse code character into the corresponding uppercase character.

Example: decode_char(".-") returns "A".

decode_word method: Decodes an entire word in Morse code, returning the string representation.

Example: decode_word("-- -.--") returns "MY".

decode method: Decodes the entire message in Morse code, providing the string representation.

Example: decode("-- -.--   -. .- -- .") returns "MY NAME".

Key Implementation Details:

Emphasized code reusability to ensure a Don't Repeat Yourself (DRY) approach.
Utilized separate methods for decoding characters, words, and entire messages.

Morse Code Message Decoding:
Applied the implemented methods to decode the Morse code message from the provided example:
.- -... --- -..- ..-. ..- .-.. .-.. --- ..-. .-. ..- -... .. . ...

Thank you @1GORDON for your review and feedback.


Implement new empowerment attribute dv_acceptability

[ ] add new attribute to people and add its default
[ ] add data to kenya.py
[ ] add functions to ingest & preprocess data
[ ] initialise attribute
[ ] record attribute
[ ] visualise attribute

PMA does not collect sufficient data on this to allow for inclusion in the model. At this time, we won't include it.


Added a wiki entry to describe postprocessing the output to filter out comments using grep

see https://github.com/MartinBoswell/fmpvc/wiki/Using-grep-to-filter-out-comments

If you add labels to the issue section you can seperate non bug like this out.


fix(internal): rename project endpoint

small PR to allow renaming a project from the client for internal use only

Codecov Report

Merging #762 (ade4527) into main (d3f3a5f) will decrease coverage by 0.11%.
Report is 1 commits behind head on main.
The diff coverage is 7.14%.

:exclamation: Current head ade4527 differs from pull request most recent head a9321d3. Consider uploading reports for the commit a9321d3 to get more accurate results

@@            Coverage Diff             @@
##             main     #762      +/-   ##
==========================================
- Coverage   88.58%   88.48%   -0.11%
==========================================
  Files         175      175
  Lines       14402    14415      +13
==========================================
- Hits        12758    12755       -3
- Misses       1644     1660      +16

Files
Coverage Δ

dataquality/internal.py
0.00% <0.00%> (ø)

dataquality/clients/api.py
49.85% <9.09%> (-1.63%)
:arrow_down:

... and 2 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


CustomNotSoSerialIT failing on Windows

Tests in error:
CustomNotSoSerialIT.before:50 » InvalidPath Illegal char <:> at index 2: /C:/U...
CustomNotSoSerialIT.after:56 » NullPointer

Btw, have you also tested that the agent actually loads on Windows?

I have not, I only fixed the failing test. I guessed some of the integration tests did that?

Aha. Yes. But they do it using attach. So a command line specified agent will execute a slightly different code path.

Maybe we can open a separate issue for testing that? This issue is fixed, and tested on Windows.


Critical: NullPointerException when trying to send a message

Create EmailMessage
Execute message.send();
Result: NullPointerException
The reason: line 731 of
EwsUtilities: cls.isAssignableFrom(Number.class) return false if cls is Integer.class

StackTrace:

Caused by: java.lang.NullPointerException
	at microsoft.exchange.webservices.data.core.ExchangeServerInfo.parse(ExchangeServerInfo.java:80)
	at microsoft.exchange.webservices.data.core.request.ServiceRequestBase.readSoapHeader(ServiceRequestBase.java:530)
	at microsoft.exchange.webservices.data.core.request.ServiceRequestBase.readResponse(ServiceRequestBase.java:485)
	at microsoft.exchange.webservices.data.core.request.ServiceRequestBase.readResponse(ServiceRequestBase.java:453)
	at microsoft.exchange.webservices.data.core.request.SimpleServiceRequestBase.internalExecute(SimpleServiceRequestBase.java:69)
	... 87 more

will be fixed with ref. 293

@tsoroka since the possible fix is merged. Plz report back if the error still exists.

closing this for now

@serious6 could you please let me know when this issue will be avaliable in a new shapshot build?

its included in the latest snapshot.


Remove strict mode for create apiview

strict mode is causing failures in language pipeline if any of the helper functions has unset properties. This is PR is a temporary work around to unblock java pipelines.

Can you point me at an example failure? In general if strict-mode causes failures it means there are potential lurking bugs and we should try to fix those if possible.

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow

@weshaggard seems to be hit in go as well: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3607285&view=logs&j=b70e5e73-bbb6-5567-0939-8415943fadb9&t=d34cfe88-f3e1-5ca1-ec50-18a5cea1099a
CC @chlowell

https://github.com/Azure/azure-sdk-for-java/blob/main/eng/scripts/Language-Settings.ps1#L658

This PR is to temporarily unblock any API review failures for Java and similarly impacted languages. .Net is working fine.
I have submitted a PR to fix the issue in Java repo. I will fix in Go also.

@benbp  @weshaggard  Here is the PR to fix the issue in Java repo. https://github.com/Azure/azure-sdk-for-java/pull/39276/files

PR to fix issue in Go: https://github.com/Azure/azure-sdk-for-go/pull/22612


Add ucvm2mesh_mpi_layer doc

the ucvm2mesh_mpi_layer code was added to support creating and merging very large meshes. We should add some basic documentation on what these new codes do, and how to use them. This code may be useful for the discontinuous meshing needed in the future.

It is documented in the wiki and also in the example/programs/mesh
Mei

From: Philip Maechling<EMAIL_ADDRESS>Sent: Tuesday, June 18, 2019 9:32 AM
To: SCECcode/UCVMC
Cc: Subscribed
Subject: [SCECcode/UCVMC] Add ucvm2mesh_mpi_layer doc (#110)
the ucvm2mesh_mpi_layer code was added to support creating and merging very large meshes. We should add some basic documentation on what these new codes do, and how to use them. This code may be useful for the discontinuous meshing needed in the future.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_SCECcode_UCVMC_issues_110-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DACE7UFZONBXLP34MIHYE5A3P3EE2DA5CNFSM4HZB2232YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G2GEZYA&d=DwMCaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=hCwk_stuE2kiGsjuxw7tlQ&m=ETZTJmHTontacZmoY8JXfC2q2hD9aPWWgN9dFajVEEE&s=zCylEWnQdlCtnn-jSlkzB8CM9gELDCD2D8V-A8fCid0&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACE7UF65LJPL3RYLQMTRJ3TP3EE2DANCNFSM4HZB223Q&d=DwMCaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=hCwk_stuE2kiGsjuxw7tlQ&m=ETZTJmHTontacZmoY8JXfC2q2hD9aPWWgN9dFajVEEE&s=K_LVkZq-BmJ0BcWDhSdK2cJUOKy67duj_C6eHpiaSFo&e=.


Communicate with downstream envoy behind private network

Title: Is it possible from the downstream envoy to establish a connection with the upstream, from the upstream service to send a tcp request to a service behind the downstream envoy without exposing the downstream envoy to a public IP/Port.
Description:
In my scenario, I have a central envoy (Upstream) that contains a management service. This management service is meant to connect to multiple clusters. Each cluster is behind a private network with the only option engress with no ingress option for envoy.
This StackOverFlow describes the senario very well:

For a hybrid-cloud use-case we are looking into the suitability of EnvoyProxy to act as a solution to move data across an on-premise firewall. The intended setup is as follows:
App A is located in an on premise network with no direct outbound or inbound Internet connection
App B is located on the cloud
An envoy proxy (PC) is placed in the cloud
An envoy proxy (PA) is placed in the on-premise network, and configured to allow outbound network connection to PC
PA creates a open bi-directional authenticated TLS connection to PC, effectively creating a tunnel between them
App B invokes an API endpoint at PC, which gets routed to PA over the open TLS connection, and forwarded by PA to App A
Routing data with Envoy to upstream clusters is well documented. However, we are wondering if Envoy is able to setup a TLS connection between two proxy instances and use that channel in both directions. The constraint is that this TLS connection can be setup from one direction (outbound) only.

We want the management service to make requests to applications over HTTP and TCP.
I have been looking into the TCPProxy option that is described in the video in stackOverflow but I am struggling to find the target needed to communicate with the downstream server.
I have also been following the examples in the Sandbox documentation but my issue is that I am not looking to setup a streaming connection but a way to tunnel the two networks so their accessable.
I have been struggling with the understanding due to limited explaination in the docs.
I have also been following the examples in the Sandbox documentation but my issue is that I am not looking to setup a streaming connection but a way to tunnel the two networks so their accessable.
I have been struggling with the understanding due to limited explaination in the docs.
Things I have looked into are and tried a few of the sandboxes examples they give.
WebSocket is a one to one streaming and does not allow reverse proxy to multiple services.
I have read that Envoy Connect/TCPProxy could be a solution, I tried this but struggled to find an endpoint to the DownStream Cluster that is connected to the listener on the Upstream server.
Any advise would be highly apprichiated.
[optional Relevant Links:]

StackOverflow

Hi, could you provide some arch graph to help others to understand you issue? It seems that is complicated.

Hello, Thank you for asking for more clarity. I have attached an image to show the direction of envoy traffic and the end goal.
Please let me know if you have any further questions.

As far as I know, it would be impossible. May be we can implement some special extensions to achieve this, but I personally don't think it's proper way of using Envoy. May be I miss something.
cc @mattklein123 @alyssawilk

Hello @wbpcode
I would have to disagree with you. In the envoy documentation Envoy speak about how the solution would replace stunnel.
In the sTunnel example it is possible to establish a Client to Server Tunnel with the Server only having to expose its port publically. Once the tunnel is up, you can specify to expose services and ports over that tunnel. In essense, creating a sub-network between the two networks.
If we look at WireGuard for example, again we can have a client to server connection that allows the exposure of some or all the network.
I feel that if there is a tunnel between the two open like webSocket does, then having service discovery or just port exposure should be a small step of this type of implementation.
I have worked in Finance, Telecommunication and Military Enterprise organisations and this style of central management to reduce public internet exposure is a must.
As for the future, statistics show that the future is Hybrid Cloud. This is a great example to utlise that scenario.
Maybe my opinion is Bias ;)

In the envoy documentation Envoy speak about how the solution would replace stunnel.

May be this is what I missed. Could you give a doc link about this?

Hey, well its dependent on on how you inturpt the documentation.
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/listeners/tcp_proxy

The TCP proxy filter performs basic 1:1 network connection proxy between downstream clients and upstream clusters. It can be used by itself as an stunnel replacement

But then there is no documentation explaining this.

Another Example of Wireguard doing something almost similiar:
https://iliasa.eu/wireguard-how-to-access-a-peers-local-network/

Yeah, it's possible create bi-direction TCP connections by the TCP proxy. It means that the App B can create a connection to App A and then the App A can send anything to the App B proactively.
But I don't think this is what you want.
From you description, you may want create a tunnel connection between two envoy, and then, the App A can send the request to App B in any time just by create a new connection with the left envoy. Did I understand correctly?

On your second point:

From you description, you may want create a tunnel connection between two envoy proxies, and then, the App A can send the request to App B in any time just by create a new connection with the left envoy. Did I understand correctly?

Yes! exactly. So setting up a Downstream (Right Envoy) to Upstream (Left Envoy) tunnel seems straight forward and I don't seen any issue here.
But the part that is confusing me is the "and then" part of your statement. When App A communicates with the Upstream envoy (Left Envoy) how does this work?
My best guess would be that App A would communicate an a seperate listener from the tunnel, When it connects and listener filter points to a cluster, what would be the target endpoint to target the tunnel connection that is already established so I can forward the listener request the Downstream envoy (Right Envoy)?

So to elaborate more on my local env:
Downstream Envoy (Right Envoy):
admin:
  address:
    socket_address:
      protocol: TCP
      address: <IP_ADDRESS>
      port_value: 9902
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: <IP_ADDRESS>
        port_value: 10001
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: ingress_http
          cluster: cluster_0
  - name: listener_1
    address:
      socket_address:
        address: <IP_ADDRESS>
        port_value: 10002
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: ingress_http
          cluster: cluster_1
  clusters:
  - name: cluster_0
    connect_timeout: 5s
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <IP_ADDRESS>
                port_value: 10000
  - name: cluster_1
    connect_timeout: 5s
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <IP_ADDRESS>
                port_value: 80

So from the Right Envoy we are connecting to our UpStream Envoy server via 10000 on listener_0. This is establishing our TCPProxy Tunnel.
Listener_1 is intended when the envoy Upstream returns  a request via the tunnel.
Upstream Envoy (Left Envoy):
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: <IP_ADDRESS>
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: ingress_http

Missing pieces here are, Upstream Envoy requires a Cluster for TcpProxy.
But in this regard I dont want to request to go any further. I want to tunnel to wait for an application to make a request to the Upstream envoy which will be sent to the downstream envoy and target listener_1

My best guess would be that App A would communicate an a seperate listener from the tunnel, When it connects and listener filter points to a cluster, what would be the target endpoint to target the tunnel connection that is already established so I can forward the listener request, from the upstream envoy (left envoy) to the Downstream envoy (Right Envoy)?

This is also what i guess. But it's not supported by envoy for now. May be it's possible to implement it with a special extension, but in my personal opinion, I don't think envoy can do better than professional tunneling software in this scene.

From the Downstream Envoy (Right Envoy) It would be just a cluster endpoint too App B.

Another extension is necessary here. The request from App A will be treat as response by the Right envoy, so, we need additional cast and route here.

@wbpcode I apprichiate you taking your time to work through this issue.
I am going to follow up to make a Feature request. As a regular around here, any tips you can give me in formatting the feature request so its easy for the community here to digest?
Again! thanks for your time.

I am going to follow up to make a Feature request. As a regular around here, any tips you can give me in formatting the feature request so its easy for the community here to digest?

Take it easy. Just describe you requirement more clearly. If is necessary, a architecture graph would be better. And specific to your current requirement, I think you may need prepare some very convincing reasons. Clearly, this requirement is not a typical application scenario for a proxy.

This requirement @Shinerrs has is similar to how Microsoft Azure (i have experience with) has a proxy which is put on-premises and makes outbound tcp connection to the central proxy in the cloud. The idea being the on-premises makes outbound connections and blocks inbound which is great from security point of view. Then an App in the cloud can communicate TO on-premises networks but the on-premises network is the initiator of the secure channel.
In my environment i need a proxy (preferably Envoy, as i use it a lot) to proxy connections from a Grafana instance in the Cloud TO datasources located on-premises but instead of Cloud---->Datacenter the connection is pre-established Datacenter---->Cloud.  Exactly the same desired traffic direction described on the image a few posts up.

@wbpcode I apprichiate you taking your time to work through this issue.
I am going to follow up to make a Feature request. As a regular around here, any tips you can give me in formatting the feature request so its easy for the community here to digest?
Again! thanks for your time.

Hi @Shinerrs may I ask if you are going to raise a feature request for this?

We also have some interest in this for the same reasons mentioned by @Shinerrs and @pxpnetworks.
We are searching for a kind of L7 HTTP double proxy setup with Envoy, with open filtering capabilities, where a reverse tunnel (from private network to public cloud) is created between between proxies to convey HTTP requests/responses and overcome private network not accepting inbound traffic.
It looks like it is not possible yet solely with Envoy. I've looked to other popular proxies like Traefik and it also looks unsupported. Teleport looks interesting but doesn't seem to have the level of filtering capabilities in Envoy or Traefik.
Any recommendation? Or maybe this feature request is about our use case too?

I made a related post on the Envoy discussion forum at https://groups.google.com/g/envoy-users/c/rE3PfA7kgyQ/m/UrnMA2SaBwAJ . It doesn't look likely to be possible without a probably-complex plugin.
From what I can tell it's probably not possible for the server side to initiate a new HTTP/2 stream, so the server end has to exchange command-and-control messages with the client to ask the client to initiate new streams, or the client and server have to use their own multiplexing layer on top of HTTP/2's own. HTTP/3 doesn't seem to change this. I'd love to be wrong, if anyone can show me how the spec permits it. The only open source tool I found that implements HTTP reverse tunnels is https://github.com/hotnops/gtunnel and from my reading of the code it seems to use the command-and-control messaging strategy.
If I'm right about that (which is far from certain) the Envoy plugin would have to expose a custom Envoy listener that, when connected to, replied on the inbound tunnel's socket to request a new HTTP/2 CONNECT stream to transit the outbound traffic on. And a corresponding plugin on the initiator Envoy end would need to know how to accept that tunnelled traffic and proxy it to regular endpoints then tunnel the reply traffic back.
I'd love to be wrong though.

@fab-mindflow Did you ever find a satisfactory solution to that issue? As noted above, I'm looking for something along the same lines and have found astonishingly slim pickings. See related Istio forums post too.

Hi!
We have proposed a design solution for this issue, do check out:
https://github.com/envoyproxy/envoy/issues/33320


Add pointermove, pointerdown, pointerout, and pointerup events to the…

… interaction manager
To enable multi-touch to work in IE 11 and other browsers that pointer events.
From pointermove, pointerdown, pointerout, and pointerup browser events; the following displayObject events are available:
pointerdown, pointerup, pointerupoutside, pointerover, pointerout, pointercommanddown, pointercommandup, pointercommandtap, pointertap, pointercommandupoutside, pointerupoutside

I'm a fan of the pointer event implementation, but why maintain 2 virtual objects, 1 for the mouse and 1 for the pointer? Wouldn't it makes sense to abstract everything to pointer events, e.g. mouse events and touch events?

Also you should take note of this issue https://github.com/pixijs/pixi.js/issues/2071 by @englercj
I for one would think a rewrite on pointer events would be one option.
But I am also biased toward a proposal I made a year or two ago where pixi does not attempt to maintain any event listeners whatsoever. It should simply contain helper methods to transform and project screen/page/client/element coordinates to pixi coordinates, and apply a hittest to the pixi object tree. The hittest can be abstracted further to apply a particular virtual pixi event to the tree. In this case, you would still need the appropriate event abstraction to go with, and like I said, I'm a fan of pointer events ;)

Think of something like this:
var point = PIXI.interaction.pointFromDomEvent(domEvent, containerObject); // containerObject is pixi scope
PIXI.interaction.applyEventAtPoint("pointerdown", point, containerObject); // containerObject is pixi scope

Ok probably not the best place ot post these comments, more appropriate on #2071

Good feedback. While it would be good to abstract everything to pointer events as suggested, I see this solution as a intermediate implementation until issue #2071 is done. I also see your point that it would be best for PIXI to not handle the events. Initially i didn't want to modify the PIXI source code, but it turned out to be cleaner than duplicating/implementing my own hit testing code.

Could this be merged in?

@GoodBoyDigital Can you take a look at this? I don't have a touch device to test with.

Is there an eta on this? If it's going to be while, I'll just use my forked repo; but I'd rather not if I don't have to.

Closing in favor of #2476.