kube-aws v0.9.6-rc.2 - Unstable Etcd3 cluster

Hello, we have been testing out kube-aws and had started a cluster using kube-aws v0.9.6-rc.2  started on Apr 17 15:46:43 GMT-600 2017.  Since then, we have seen multiple failures & degradation events for the Etcd3 cluster, without any action or change required to cause this.
It appears that for some reason, the AutoScalingGroup for each Etcd node is reporting errors similar to the following:

Description:
Terminating EC2 instance: i-01162861a56c5fb21

Cause:
At 2017-05-03T00:39:05Z an instance was taken out of service in response to a EC2 instance status checks failure.

This has happened a couple times for each node, resulting in times when with a 3 node cluster it was put into a degraded state.  After this point, kube-apiserver is unable to contact Etcd, and any changes to the cluster requiring kube-apiserver access or kubectl commands will not work.
Here are the details on each AutoScaling Group as shown in the EC2 Console:
Etcd0

Etcd1

Etcd2

Might be caused by https://github.com/kubernetes-incubator/kube-aws/issues/640

I've confirmed that this is reproducible on kube-aws v0.9.6 "stable".
Steps to reproduce with EtcD logs shipped to a Pre-existing ELK stack:

Make sure your ElasticSearch nodes are contactable within a pre-existing VPC
Use this cluster.yaml fill in your VPC & custom cluster details, follow normal kube-aws up process (using v.0.9.6 stable):

kube-aws render credentials --generate-ca
kube-aws render stack
kube-aws up --export --s3-uri s3://etcd-fail-example/
kube-aws validate  --s3-uri s3://etcd-fail-example/
kube-aws up  --s3-uri s3://etcd-fail-example/

Let cluster come up, watch logs come in to your ELK stack

Leave cluster alone doing absolutely nothing for about 3 days

Come back later to an unhealthy Etcd3 cluster

Check the ELK logs.  You will probably see:
 2017-05-31 21:16:56.742308 E | rafthttp: failed to find member 123abc456badbeef in cluster fffabc123badcafe
 ### [...SNIP...]  Many of this  ^^^^^ error repeatedly
 2017-05-31 22:17:50.356072 W | rafthttp: failed to process raft message (raft: stopped)

In AWS AutoScalingGroup history page you will most likely see "Terminating EC2 instance" for all 3 nodes at some time in the past 3 days.

Try checking EtcD3 cluster health by connecting via SSH to one of the nodes, then run:
 source /var/run/coreos/etcdadm-environment
 docker run -ti  -e ETCDCTL_API=3 -v /etc/ssl/certs:/etc/ssl/certs:ro  quay.io/coreos/etcd:v3.1.7 \
     etcdctl   --endpoints="$ETCD_ENDPOINTS" --cert=$ETCDCTL_CERT_FILE --key=$ETCDCTL_KEY_FILE --cacert=$ETCDCTL_CACERT \
                 endpoint health

 ## You will probably see:
 2017-05-31 23:02:09.548464 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
 2017-05-31 23:02:09.549244 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
 2017-05-31 23:02:09.549949 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
 https://ec2-12-123-231-12.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 59.700185ms
 https://ec2-55-66-123-231.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 164.421588ms
 https://ec2-77-88-123-1.compute-1.amazonaws.com:2379 is unhealthy: failed to connect: grpc: timed out when dialing

It appears that after this failure, the etcdadm-reconfigure.service tries to recover the cluster.  It may or may not be successful depending on the timing of the node failures & terminations.  I have had this happen where the cluster goes below quorum and is non-recoverable by the automatic process, and also where 2 nodes are still healthy.
Presumably, the probability of going below quorum decreases with increasing odd cluster size... but for bug reproduction a 3 node cluster will most likely exhibit the failure.  So far 3 out of 3 tests with fresh clusters have had the problem.

Hi @trinitronx, thank you very much for the detailed feedback 👍
First of all, please let me sync.
v0.9.6 is not a "stable" version in regard to the fact that it is the first version introduced the automatic disaster recovery feature. I'd say it just "v0.9.6 final".
Also, kube-aws is a community-maintained project. Many of its active contributors, including me, do run their production clusters with kube-aws but that doesn't mean every "final" release has every feature "stabilized". I won't say a feature "stable" until someone battle-test it.
Sorry if it missed your expectation in that regard and made you frustrated.
However, a detailed feedback like yours is SUPER HELPFUL to improve kube-aws!
Thank you very much for sharing your hard-won experience :bow:

Backing to the topic -

you will most likely see "Terminating EC2 instance" for all 3 nodes at some time in the past 3 days.

It does seem like you have affected by #640, which has been fixed since v0.9.7-rc.1.
If I could ask you for more, would you mind confirming if it actually fix your case?

@mumoshu :  Thanks for the quick response!  Ah, I suppose my assumption of "stable" was incorrect.  This is definitely good to know!  I did see that the etcd auto-recovery was just added, so I did not assume this would work 100%.  However, I did not expect the cluster to go unhealthy so fast in the first place, and was hoping not to need it ;-)
We have been running an older kubernetes cluster with etcd2 and luckily have never had to touch the etcd2 nodes at all.  I suppose this implementation is different enough, and uses etcd3, so my initial assumption based on prior experience was definitely not correct.
I think it may make sense to test out this v0.9.7-rc.1, although I had thought that this fix was in v0.9.6 (also wrong assumption).
Hopefully my reproduction & ELK logging for Etcd nodes can provide some debugging value.  Initially I thought that I would be able to debug the cluster myself, but I am seeing basically the logs I posted.
This time, the quorum was kept with 2 nodes coming back healthy.  The third node was not able to auto-recover, and is logging the rafthttp errors I mentioned above.
When I test connection to this node, there are unexplained connection errors.  I try to connect to etcd on port 2379 of the unhealthy node public hostname from $ETCD_ENDPOINTS, even from localhost or local EC2 VPC private IP on that same node.  Tried with both curl and etcdctl, same result just a connection timeout.  So this appears very odd because all nodes are still in the same SecurityGroup, and indeed no SG should apply for localhost connection.  I haven't had much experience with a deep-dive debug into Etcd, so any help on where to start there would be appreciated.
Basically what I've tried so far is in the steps I detailed above.  I also have tested out this curl command on the bad node, and on other healthy nodes:
source /var/run/coreos/etcdadm-environment
curl  -vv --cert $ETCDCTL_CERT_FILE --key $ETCDCTL_KEY_FILE --cacert $ETCDCTL_CACERT -X GET https://<IP_ADDRESS>:2379/metrics

Bad node result:
curl  -vv --cert $ETCDCTL_CERT_FILE --key $ETCDCTL_KEY_FILE --cacert $ETCDCTL_CACERT -X GET https://<IP_ADDRESS>:2379/metrics
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 2379 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca.pem
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <IP_ADDRESS>:2379
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <IP_ADDRESS>:2379

Good node result:
curl  -vv --cert $ETCDCTL_CERT_FILE --key $ETCDCTL_KEY_FILE --cacert $ETCDCTL_CACERT -X GET https://ec2-12-34-123-321.compute-1.amazonaws.com:2379/metrics
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to ec2-12-34-123-321.compute-1.amazonaws.com (<IP_ADDRESS>) port 2379 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca.pem
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=kube-aws; CN=kube-etcd
*  start date: May 23 23:27:50 2017 GMT
*  expire date: May 23 23:27:51 2018 GMT
*  subjectAltName: host "ec2-12-34-123-321.compute-1.amazonaws.com" matched cert's "*.compute-1.amazonaws.com"
*  issuer: O=kube-aws; CN=kube-ca
*  SSL certificate verify ok.
> GET /metrics HTTP/1.1
> Host: ec2-12-34-123-321.compute-1.amazonaws.com:2379
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 29606
< Content-Type: text/plain; version=0.0.4
< Date: Wed, 31 May 2017 23:47:15 GMT
<
[... SNIP ...]
[... PROMETHEUS METRICS HERE ...]

So I did dig in a bit and check these logs.  It seems that the recovery service etcdadm-reconfigure is trying to recover, but is stuck on an S3 access denied error.
Here is a Gist with scrubbed output from etcdadm-reconfigure JournalD logs.  Basically in summary it is trying to recover by deleting the data dir, then failing with:
An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

SystemD Unit "status" output
core@ip-10-100-2-111 ~ $ systemctl status etcdadm-reconfigure
● etcdadm-reconfigure.service - etcdadm reconfigure runner
   Loaded: loaded (/etc/systemd/system/etcdadm-reconfigure.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2017-05-31 22:17:44 UTC; 2h 25min ago
  Process: 29453 ExecStartPost=/usr/bin/systemctl start etcdadm-check.timer (code=exited, status=0/SUCCESS)
 Main PID: 28736 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 32768)
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/etcdadm-reconfigure.service

May 31 22:17:42 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_reconfigure: cluster is already healthy but still in bootstrap process after the disaster recovery. searching for a etcd snapshot to rec
May 31 22:17:42 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: checking existence of s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7
May 31 22:17:43 ip-10-100-2-111.ec2.internal etcdadm[28736]: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
May 31 22:17:44 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etc
May 31 22:17:44 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_bootstrap: remote snapshot for etcd1 does not exist. skipped downloading
May 31 22:17:44 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_local_snapshot_exists: checking existence of file /var/run/coreos/etcdadm/snapshots/etcd1.db
May 31 22:17:44 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_bootstrap: backup not found. starting brand new etcd1...
May 31 22:17:44 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_set_initial_cluster_state: setting initial cluster state to: new
May 31 22:17:44 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: _systemctl_daemon_reload: running `systemctl daemon-reload` to reload etcd-member.service
May 31 22:17:44 ip-10-100-2-111.ec2.internal systemd[1]: Started etcdadm reconfigure runner.

core@ip-10-100-2-111 ~ $ systemctl status etcd-member
● etcd-member.service - etcd (System Application Container)
   Loaded: loaded (/usr/lib/systemd/system/etcd-member.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/etcd-member.service.d
           └─20-aws-cluster.conf, 30-unit-type.conf, 40-version.conf
   Active: activating (start) since Wed 2017-05-31 22:17:44 UTC; 2h 28min ago
     Docs: https://github.com/coreos/etcd
  Process: 29494 ExecStartPre=/usr/bin/chown -R etcd:etcd /var/lib/etcd2 (code=exited, status=0/SUCCESS)
  Process: 29490 ExecStartPre=/usr/bin/systemctl is-active decrypt-assets.service (code=exited, status=0/SUCCESS)
  Process: 29487 ExecStartPre=/usr/bin/systemctl is-active cfn-etcd-environment.service (code=exited, status=0/SUCCESS)
  Process: 29466 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid (code=exited, status=0/SUCCESS)
  Process: 29456 ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos (code=exited, status=0/SUCCESS)
 Main PID: 29496 (etcd)
    Tasks: 6 (limit: 32768)
   Memory: 30.9M
      CPU: 29min 2.035s
   CGroup: /system.slice/etcd-member.service
           └─29496 /usr/local/bin/etcd

Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.700705 E | rafthttp: failed to find member 123abc456badbeef in cluster fffabc123badcafe
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.702247 E | rafthttp: failed to find member 123abc456badbeef in cluster fffabc123badcafe
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.703720 W | rafthttp: failed to process raft message (raft: stopped)
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.836365 W | rafthttp: failed to process raft message (raft: stopped)
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.836583 E | rafthttp: failed to find member 5555abcdef000000 in cluster fffabc123badcafe
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.836678 E | rafthttp: failed to find member 5555abcdef000000 in cluster fffabc123badcafe
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.884561 E | rafthttp: failed to find member 123abc456badbeef in cluster fffabc123badcafe
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.893164 E | rafthttp: failed to find member 123abc456badbeef in cluster fffabc123badcafe
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.894621 W | rafthttp: failed to process raft message (raft: stopped)
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.895131 W | rafthttp: failed to process raft message (raft: stopped)
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.970338 E | rafthttp: failed to find member 5555abcdef000000 in cluster fffabc123badcafe
Jun 01 00:46:11 ip-10-100-2-111.ec2.internal etcd-wrapper[29496]: 2017-06-01 00:46:11.971201 E | rafthttp: failed to find member 5555abcdef000000 in cluster fffabc123badcafe

Conclusion:
etcd-member service is up, but in a non-recovered state due to S3 permission error when etcdadm-reconfigure tried to download the snapshot backup.
So, etcd is indeed up but somehow in a bad state.  Additionally, connecting to the API on this unhealthy node is not working.
More Etcd connection debugging
On the bad node, etcd is supposedly listening on that port:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp      129      0 <IP_ADDRESS>:2379        <IP_ADDRESS>:*               LISTEN      29496/etcd
tcp        0      0 <IP_ADDRESS>:2380        <IP_ADDRESS>:*               LISTEN      29496/etcd

Does not seem to respond to a telnet connect either:
core@ip-10-100-2-123 ~ $ /usr/bin/toolbox
Downloading sha256:2bf01635e2a [=============================] 72.9 MB / 72.9 MB
successfully removed aci for image: "sha512-42c13efebbc9ba9bd6324f07293c6eadb424042564821024eca129748e8f6054"
rm: 1 image(s) successfully removed
Spawning container core-fedora-24 on /var/lib/toolbox/core-fedora-24.
Press ^] three times within 1s to kill container.
[root@ip-10-100-2-123 ~]#
[root@ip-10-100-2-123 ~]#
[root@ip-10-100-2-123 ~]# dnf -y install telnet
Fedora 24 - x86_64 - Updates                                      22 MB/s |  23 MB     00:01
Fedora 24 - x86_64                                                23 MB/s |  47 MB     00:02
Last metadata expiration check: 0:01:06 ago on Thu Jun  1 00:10:21 2017.
Dependencies resolved.
=================================================================================================
 Package             Arch                Version                      Repository            Size
=================================================================================================
Installing:
 telnet              x86_64              1:0.17-66.fc24               updates               67 k

Transaction Summary
=================================================================================================
Install  1 Package

Total download size: 67 k
Installed size: 113 k
Downloading Packages:
telnet-0.17-66.fc24.x86_64.rpm                                   496 kB/s |  67 kB     00:00
-------------------------------------------------------------------------------------------------
Total                                                            171 kB/s |  67 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Installing  : telnet-1:0.17-66.fc24.x86_64                                                 1/1
  Verifying   : telnet-1:0.17-66.fc24.x86_64                                                 1/1

Installed:
  telnet.x86_64 1:0.17-66.fc24

Complete!

[root@ip-10-100-2-123 ~]#
[root@ip-10-100-2-123 ~]#
[root@ip-10-100-2-123 ~]#  telnet <IP_ADDRESS> 2379
Trying <IP_ADDRESS>...
Connected to <IP_ADDRESS>.
Escape character is '^]'.
GET /

[... NO RESPONSE ...]
[... Maybe only SSL handshake is expected? ...]
[... Ctrl+D ...]

core@ip-10-100-2-123 ~ $ echo 'TOOLBOX_BIND="--bind=/:/media/root --bind=/usr:/media/root/usr --bind=/run:/media/root/run --bind=/etc/ssl/certs:/etc/ssl/certs"' >> .toolboxrc
core@ip-10-100-2-123 ~ $ toolbox
[root@ip-10-100-2-123 ~]# dnf -yq install openssl

[root@ip-10-100-2-123 ~]# openssl s_client -connect <IP_ADDRESS>:2379 -CAfile /etc/ssl/certs/ca.pem -cert /etc/ssl/certs/etcd-client.pem -key /etc/ssl/certs/etcd-client-key.pem
CONNECTED(00000003)
GET /

write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 201 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time:<PHONE_NUMBER>
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Additional information
I do see symptoms of #640 when I list exited docker containers I see many exited old awscli containers:
core@ip-10-100-2-123 ~ $ sudo docker ps -a  | grep -c awscli
3369

@mumoshu :  Perhaps last bit of helpful info:
On the bad etcd node, it has an IAM Role with this policy attached (private info scrubbed):
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:<AWS_REGION_HERE>:<AWS_ACCOUNT_ID_HERE>:key/<KMS_KEY_ID_HERE>",
      "Effect": "Allow"
    },
    {
      "Action": "ec2:DescribeTags",
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": "ec2:DescribeVolumes",
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": "ec2:AttachVolume",
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": "ec2:DescribeVolumeStatus",
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": "ec2:AssociateAddress",
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::test-etcd-fail-example/kube-aws/clusters/etcd-fail/exported/stacks/control-plane/userdata-etcd*",
      "Effect": "Allow"
    },
    {
      "Condition": {
        "StringLike": {
          "s3:prefix": "/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/*"
        }
      },
      "Action": [
        "s3:List*"
      ],
      "Resource": "arn:aws:s3:::test-etcd-fail-example",
      "Effect": "Allow"
    },
    {
      "Action": [
        "s3:*"
      ],
      "Resource": "arn:aws:s3:::test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/*",
      "Effect": "Allow"
    },
    {
      "Action": "ec2:DescribeInstances",
      "Resource": "*",
      "Effect": "Allow"
    }
  ]
}

Further Debugging
Today I was able to dig deeper into the S3 issue.  I was able to find the docker run command that was running awscli with the following:
source /var/run/coreos/etcdadm-environment
export $(cat  /var/run/coreos/etcdadm-environment  /etc/etcd-environment | sed -e 's/\([A-Z_0-9]+\{0,\}\)=\(.*\)/\1/g'  | sort | uniq | tr '\n' ' ')
sudo -E  /opt/bin/etcdadm  _docker_awscli_command | tr '\n' ' '

I then ran that command on my system using:
docker run -e AWS_DEFAULT_REGION=us-east-1 --net=host --volume /var/run/coreos:/var/run/coreos quay.io/coreos/awscli aws s3 ls s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

The output in debug mode shows that the IAM Role for this node is being used:
$ docker run -e AWS_DEFAULT_REGION=us-east-1 --net=host --volume /var/run/coreos:/var/run/coreos quay.io/coreos/awscli aws s3 ls s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/ --debug

2017-06-03 00:38:32,376 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.15 Python/2.7.12 Linux/4.9.24-coreos botocore/1.4.72
2017-06-03 00:38:32,376 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', 's3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/', '--debug']
2017-06-03 00:38:32,377 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f8314242a28>
2017-06-03 00:38:32,377 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f83146468c0>
2017-06-03 00:38:32,377 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2017-06-03 00:38:32,377 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f83141cbc08>
2017-06-03 00:38:32,378 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x7f831466d050>
2017-06-03 00:38:32,378 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x7f83141cbc08>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <function uri_param at 0x7f831466d050>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <function uri_param at 0x7f831466d050>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f8314624d10>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <function uri_param at 0x7f831466d050>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <function uri_param at 0x7f831466d050>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <function uri_param at 0x7f831466d050>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <function uri_param at 0x7f831466d050>
2017-06-03 00:38:32,379 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f8314624d10>
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2017-06-03 00:38:32,380 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2017-06-03 00:38:32,395 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): <IP_ADDRESS>
2017-06-03 00:38:32,396 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 60
2017-06-03 00:38:32,398 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): <IP_ADDRESS>
2017-06-03 00:38:32,398 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/etcd-fail-Controlplane-1ABCD12345678-IAMRoleEtcd-ABCDEFOOOOOK HTTP/1.1" 200 894
2017-06-03 00:38:32,399 - MainThread - botocore.credentials - INFO - Found credentials from IAM Role: etcd-fail-Controlplane-1ABCD12345678-IAMRoleEtcd-ABCDEFOOOOOK
2017-06-03 00:38:32,400 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.7/site-packages/botocore/data/endpoints.json
2017-06-03 00:38:32,509 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.7/site-packages/botocore/data/s3/2006-03-01/service-2.json
2017-06-03 00:38:32,690 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2017-06-03 00:38:32,720 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f8314f13500>
2017-06-03 00:38:32,720 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f8314f0cc08>
2017-06-03 00:38:32,720 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-06-03 00:38:32,751 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2017-06-03 00:38:32,752 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2017-06-03 00:38:32,755 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.7/site-packages/botocore/data/s3/2006-03-01/paginators-1.json
2017-06-03 00:38:32,756 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjects: calling handler <function set_list_objects_encoding_type_url at 0x7f8314f43410>
2017-06-03 00:38:32,756 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjects: calling handler <function validate_bucket_name at 0x7f8314f40500>
2017-06-03 00:38:32,781 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjects: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f8313d52ed0>>
2017-06-03 00:38:32,782 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjects: calling handler <function add_expect_header at 0x7f8314f40938>
2017-06-03 00:38:32,782 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjects: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f8313d52ed0>>
2017-06-03 00:38:32,782 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjects) (verify_ssl=True) with params: {'body': '', 'url': u'https://s3.amazonaws.com/test-etcd-fail-example?delimiter=%2F&prefix=kube-aws%2Fclusters%2Fetcd-fail%2Finstances%2F9f6c6e20-400f-11e7-8527-500c32cd4462%2Fetcd-snapshots%2F&encoding-type=url', 'headers': {'User-Agent': 'aws-cli/1.11.15 Python/2.7.12 Linux/4.9.24-coreos botocore/1.4.72'}, 'context': {'encoding_type_auto_set': True, 'client_region': 'us-east-1', 'signing': {'bucket': u'test-etcd-fail-example'}, 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7f8313d52a10>}, 'query_string': {u'delimiter': '/', u'prefix': u'kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/', u'encoding-type': 'url'}, 'url_path': u'/test-etcd-fail-example', 'method': u'GET'}
2017-06-03 00:38:32,783 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjects: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8313d52990>>
2017-06-03 00:38:32,783 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjects: calling handler <function fix_s3_host at 0x7f831547e488>
2017-06-03 00:38:32,783 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/test-etcd-fail-example?delimiter=%2F&prefix=kube-aws%2Fclusters%2Fetcd-fail%2Finstances%2F9f6c6e20-400f-11e7-8527-500c32cd4462%2Fetcd-snapshots%2F&encoding-type=url
2017-06-03 00:38:32,783 - MainThread - botocore.utils - DEBUG - URI updated to: https://test-etcd-fail-example.s3.amazonaws.com/?delimiter=%2F&prefix=kube-aws%2Fclusters%2Fetcd-fail%2Finstances%2F9f6c6e20-400f-11e7-8527-500c32cd4462%2Fetcd-snapshots%2F&encoding-type=url
2017-06-03 00:38:32,783 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2017-06-03 00:38:32,783 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2017-06-03 00:38:32,783 - MainThread - botocore.auth - DEBUG - StringToSign:
GET

Sat, 03 Jun 2017 00:38:32 GMT
x-amz-security-token:[... Base64 Encoded signature SCRUBBED ...]
/test-etcd-fail-example/
2017-06-03 00:38:32,784 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2017-06-03 00:38:32,784 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): test-etcd-fail-example.s3.amazonaws.com
2017-06-03 00:38:32,884 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /?delimiter=%2F&prefix=kube-aws%2Fclusters%2Fetcd-fail%2Finstances%2F9f6c6e20-400f-11e7-8527-500c32cd4462%2Fetcd-snapshots%2F&encoding-type=url HTTP/1.1" 403 None
2017-06-03 00:38:32,885 - MainThread - botocore.parsers - DEBUG - Response headers: {'date': 'Sat, 03 Jun 2017 00:38:31 GMT', 'x-amz-id-2': '[... SCRUBBED ...]', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '12345678ABCDEFGH', 'x-amz-bucket-region': 'us-east-1', 'content-type': 'application/xml'}
2017-06-03 00:38:32,885 - MainThread - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>12345678ABCDEFGH</RequestId><HostId>[... SCRUBBED ...]</HostId></Error>
2017-06-03 00:38:32,885 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjects: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8313a98610>
2017-06-03 00:38:32,885 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2017-06-03 00:38:32,885 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjects: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f8313d52ed0>>
2017-06-03 00:38:32,885 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjects: calling handler <function decode_list_object at 0x7f8314f43488>
2017-06-03 00:38:32,886 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjects: calling handler <function enhance_error_msg at 0x7f8314242758>
2017-06-03 00:38:32,886 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 186, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/lib/python2.7/site-packages/awscli/customizations/commands.py", line 190, in __call__
    parsed_globals)
  File "/usr/lib/python2.7/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/usr/lib/python2.7/site-packages/awscli/customizations/s3/subcommands.py", line 472, in _run_main
    bucket, key, parsed_args.page_size, parsed_args.request_payer)
  File "/usr/lib/python2.7/site-packages/awscli/customizations/s3/subcommands.py", line 499, in _list_all_objects
    for response_data in iterator:
  File "/usr/lib/python2.7/site-packages/botocore/paginate.py", line 102, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/paginate.py", line 174, in _make_request
    return self._method(**current_kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 537, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
2017-06-03 00:38:32,893 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

Testing Solution
So for some reason the default S3 bucket policy is not letting this instance read from the bucket.
I experimented by adding a new more permissive policy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:<AWS_REGION_HERE>:<AWS_ACCOUNT_ID_HERE>:key/<KMS_KEY_ID_HERE>",
            "Effect": "Allow"
        },
        {
            "Action": "ec2:DescribeTags",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "ec2:DescribeVolumes",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "ec2:AttachVolume",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "ec2:DescribeVolumeStatus",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "ec2:AssociateAddress",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::test-etcd-fail-example/kube-aws/clusters/etcd-fail/exported/stacks/control-plane/userdata-etcd*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:List*",
                "s3:GetObject*",
                "s3:PutObject*"
            ],
            "Resource": "arn:aws:s3:::test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::test-etcd-fail-example",
            "Effect": "Allow"
        },
        {
            "Condition": {
                "StringLike": {
                    "s3:prefix": "/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/*"
                }
            },
            "Action": [
                "s3:List*",
                "s3:GetObject*"
            ],
            "Resource": "arn:aws:s3:::test-etcd-fail-example",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:*"
            ],
            "Resource": "arn:aws:s3:::test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/*",
            "Effect": "Allow"
        },
        {
            "Action": "ec2:DescribeInstances",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Validation
Using this new policy, the Etcd nodes are able to Read / Write to this snapshot.db file:
$ sudo journalctl -xn10 -u 'etcdadm*' -f
Jun 03 00:48:40 <IP_ADDRESS>.ec2.internal etcdadm[4586]: /opt/bin/etcdadm: info: member_upload_snapshot: uploading /var/run/coreos/etcdadm/snapshots/etcd0.db to s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db
Jun 03 00:48:42 <IP_ADDRESS>.ec2.internal etcdadm[4586]: [2.0K blob data]
Jun 03 00:48:42 <IP_ADDRESS>.ec2.internal etcdadm[4586]: [2.0K blob data]
Jun 03 00:48:42 <IP_ADDRESS>.ec2.internal etcdadm[4586]: [2.0K blob data]
Jun 03 00:48:42 <IP_ADDRESS>.ec2.internal etcdadm[4586]: [2.0K blob data]
Jun 03 00:48:43 <IP_ADDRESS>.ec2.internal etcdadm[4586]: [852B blob data]

docker run -e AWS_DEFAULT_REGION=us-east-1 --net=host --volume /var/run/coreos:/var/run/coreos quay.io/coreos/awscli aws s3 ls s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/
2017-06-03 00:47:05   42872864 snapshot.db

After rebooting all 3 nodes, 2 of them were able to restore successfully:
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal systemd[1]: Starting etcd health check...
-- Subject: Unit etcdadm-check.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcdadm-check.service has begun starting up.
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal systemctl[11309]: active
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal etcdadm[11322]: declare -x ETCDCTL_CACERT="/etc/ssl/certs/ca.pem"
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal etcdadm[11322]: declare -x ETCDCTL_CA_FILE="/etc/ssl/certs/ca.pem"
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal etcdadm[11322]: declare -x ETCDCTL_CERT="/etc/ssl/certs/etcd-client.pem"
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal etcdadm[11322]: declare -x ETCDCTL_CERT_FILE="/etc/ssl/certs/etcd-client.pem"
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal etcdadm[11322]: declare -x ETCDCTL_KEY="/etc/ssl/certs/etcd-client-key.pem"
Jun 03 00:52:17 ip-10-100-2-123.ec2.internal etcdadm[11322]: declare -x ETCDCTL_KEY_FILE="/etc/ssl/certs/etcd-client-key.pem"
Jun 03 00:52:19 ip-10-100-2-123.ec2.internal etcdadm[11322]: [ 5140.543532] etcd[5]: https://ec2-12-34-123-321.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 1.633271ms
Jun 03 00:52:22 ip-10-100-2-123.ec2.internal etcdadm[11322]: [ 5142.943404] etcd[5]: https://ec2-12-34-123-321.compute-1.amazonaws.com:2379 is healthy: successfully committed proposal: took = 2.274348ms

1 Node Restore Problem
The third one did not.  Here are his logs
It seems that all 3 of the etcd nodes are storing snapshot.db to the same location:
s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7-8527-500c32cd4462/etcd-snapshots/snapshot.db

@mumoshu :  Is this intended operation of the etcdadm-save.service to write all snapshots to one file?  I'm not familiar enough with etcd and the snapshot save & restore process right now to know if this is desirable or not.
After rebooting this node a couple times and restarting all etcd* services, I can get it to attempt to restore and fail in the same way each time.  Maybe this one is confused about the member id somehow?  I'm not sure why it doesn't come back up cleanly.

I most likely run into this issue this weekend.
One of my etcd instances (the only one that survived anyway) had 20k+ stopped containers, so I manually added the --rm flag to /opt/bin/etcdadm and cleaned things up with the famous docker rm $(docker ps -qa).
Just to add more information to the table, I'll explain what happened on my end.
I ran a three-node etcd cluster, and one of the instances suddenly started to consume 100% CPU. I was unable to log into it via SSH to see what was happening, so I terminated the instance (it was probably best to enable the termination protection on the instance and reboot it, but I guess I'll never know by now).
Then, when the new instance went up, I saw all these rafthttp errors, causing the new instance to be unable to join the cluster. After many hours of trial and error, this is how I managed to restore the cluster:

logged via SSH to one of the healthy instances and ran etcdctl member list to find out the ID of the failing member, and then ran etcdctl member remove <id> (the failing member was marked with [unstarted]). I did have some trouble to run these commands due to the loss of quorum (since two out of three nodes were failing intermittently), so I kept trying until I eventually I managed to remove the failing member
edited the /etc/etcd-environment and set the ETCD_INITIAL_CLUSTER_STATE variable to "existing"
removed the contents of /var/lib/etcd2/member to ensure a clean startup. I remember trying to add the new member without doing this, but only after removing this directory I got rid of the rafthttp errors (or at least I believe so)
ran etcdctl member add etcd<N> https://<node-hostname>:2380, the new member immediately started to emit logs showing that it was able to join the cluster and started to catch up with the other nodes
waited some time for the new node to fetch all data

The step-by-step might not be accurate as I'm not an etcd expert myself, but, if I remember correctly, this is how I manage to retore two failing members on my cluster.

@danielfm: Thanks for the helpful procedure to restore etcd.  If this happens again I'll be sure to try this.

Update
I've made the changes I mentioned above to the S3 bucket policy in kube-aws here.  This appears to fix the S3 permission errors I was seeing such as:
May 31 22:17:42 ip-10-100-2-111.ec2.internal etcdadm[28736]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: checking existence of s3://test-etcd-fail-example/kube-aws/clusters/etcd-fail/instances/9f6c6e20-400f-11e7
May 31 22:17:43 ip-10-100-2-111.ec2.internal etcdadm[28736]: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

@trinitronx Thank you very much for your efforts :bow:
I'm trying to understand - so probably we had been missing an IAM policy for checking existence of snapshot.db, which resulted in inability to recover a permanently failed etcd node from the snapshot?
// Too bad if I had missed such a defect in my manual testing 😢

@trinitronx

Is this intended operation of the etcdadm-save.service to write all snapshots to one file? I'm not familiar enough with etcd and the snapshot save & restore process right now to know if this is desirable or not.

Yes. Currently, every etcd node is tried to be recovered from the same snapshot when in a disaster recovery process, so that the second and the following etcd nodes won't need to catch up huge data from the etcd leader. The bunch of comments in etcdadm_reconfigure() in the etcdadm script would help understading how it works.
Of course, any questions are always welcomed 👍

@mumoshu :  Just wanted to check back in to give some battle testing results of the cluster so far.  We have not seen any more failures for the etcd nodes in this cluster, and they have been staying up this week.
However, we are seeing some strange behavior when trying to interact with the cluster via kubectl or kubernetes-dashboard.  Basically we are seeing intermittent timeout errors for kubectl commands and kubernetes-dashboard pages.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"0480917b552be33e2dba47386e51decb1a211df6", GitTreeState:"clean", BuildDate:"2017-06-16T21:16:54Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3+coreos.0", GitCommit:"70bc0937018d89b0d529d428eed8c444d8e7729a", GitTreeState:"clean", BuildDate:"2017-05-11T18:15:57Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get pods
Error from server (ServerTimeout): the server cannot complete the requested operation at this time, try again later (get pods)

# A couple seconds later, we get a response but the pod is stuck in Pending

$ kubectl get pods
NAME                                        READY     STATUS    RESTARTS   AGE
example-pod-cronjob-test-1234567890-abcd1   0/1       Pending   0          3h

When hitting a kubernetes-dashboard page: https://example.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#!/workload?namespace=default
Gateway Timeout (504)

the server cannot complete the requested operation at this time, try again later (get pods)

This seems to be intermittent, sometimes we can hit the kube-apiserver directly from the same browser using: https://example.com/api/v1/namespaces/default/pods
When it fails, it returns:
{"metadata":{},"status":"Failure","message":"The list operation against pods could not be completed at this time, please try again.","reason":"ServerTimeout","details":{"name":"list","kind":"pods"},"code":500}

Other times, the same API endpoint returns 200 OK with results all within 300-400ms according to Google Chrome's Developer Tools -> Network request info.
So there is something still causing intermittent kube-apiserver failures, presumably also causing pods to be stuck in Pending.
Still not sure what the root cause is, but I thought I'd provide the feedback.

Update:
Seems that there is some issue with flanneld:
Jun 12 19:40:51 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:51.149236    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:52 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:52.952952    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:53 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:53.155630    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:54 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:54.961201    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:55 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:55.167600    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:56 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:56.996751    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:57 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:57.173963    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:59 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:59.004762    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:40:59 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:40:59.189276    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:01 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:01.011583    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:01 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:01.197205    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:03 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:03.019284    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:03 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:03.204723    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:05 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:05.026410    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:05 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:05.211500    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:07 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:07.033600    1451 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured
Jun 12 19:41:07 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: E0612 19:41:07.218702    1451 watch.go:272] Subnet watch failed: client: etcd cluster is unavailable or misconfigured

Solution (for now)
Going onto the master nodes and issuing: sudo systemctl restart flanneld seems to have fixed it
$  journalctl -xn -u flanneld.service

-- Subject: Unit flanneld.service has begun shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit flanneld.service has begun shutting down.
Jun 17 00:32:15 ip-10-123-1-12.ec2.internal flannel-wrapper[1451]: I0617 00:32:15.881177    1451 main.go:172] Exiting...
Jun 17 00:32:15 ip-10-123-1-12.ec2.internal rkt[11157]: "98765432-1000-ffff-1111-abcdef012345"
Jun 17 00:32:15 ip-10-123-1-12.ec2.internal systemd[1]: Stopped flannel - Network fabric for containers (System Application Container).
-- Subject: Unit flanneld.service has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit flanneld.service has finished shutting down.
Jun 17 00:32:15 ip-10-123-1-12.ec2.internal systemd[1]: Starting flannel - Network fabric for containers (System Application Container)...
-- Subject: Unit flanneld.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit flanneld.service has begun starting up.

-- Subject: Unit flanneld.service has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit flanneld.service has finished starting up.
--
-- The start-up result is done.
Jun 17 00:32:33 ip-10-123-1-12.ec2.internal flannel-wrapper[11405]: I0617 00:32:33.830637   11405 network.go:58] Watching for L3 misses
Jun 17 00:32:33 ip-10-123-1-12.ec2.internal flannel-wrapper[11405]: I0617 00:32:33.831017   11405 network.go:66] Watching for new subnet leases

I've noticed that with our cluster there was also a problem with kube-proxy not being able to hit the apiserver.  Presumably this was due to broken flanneld.
Here were the error messages in container logs for kube-proxy:
Worker nodes
$ docker logs --tail=100 -f  $(docker ps -a | grep kube-proxy | grep -v pause | awk '{ print $1 }')

E0612 20:30:34.634628       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: the server cannot complete the requested operation at this time, try again later (get endpoints)
E0612 20:31:35.638726       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: the server cannot complete the requested operation at this time, try again later (get endpoints)
E0612 20:32:36.641849       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: the server cannot complete the requested operation at this time, try again later (get endpoints)

E0612 20:47:49.031903       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: the server cannot complete the requested operation at this time, try again later (get services)
E0612 20:48:50.035194       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: the server cannot complete the requested operation at this time, try again later (get services)
E0612 20:49:51.050341       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: the server cannot complete the requested operation at this time, try again later (get services)

Controller nodes
$ docker logs --tail=100 -f  $(docker ps -a | grep kube-proxy | grep -v pause | awk '{ print $1 }')

E0617 03:29:58.726256       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get http://<IP_ADDRESS>:8080/api/v1/services?resourceVersion=0: dial tcp <IP_ADDRESS>:8080: getsockopt: connection refused
E0617 03:29:58.727810       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get http://<IP_ADDRESS>:8080/api/v1/endpoints?resourceVersion=0: dial tcp <IP_ADDRESS>:8080: getsockopt: connection refused
E0617 03:29:59.727147       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get http://<IP_ADDRESS>:8080/api/v1/services?resourceVersion=0: dial tcp <IP_ADDRESS>:8080: getsockopt: connection refused
E0617 03:29:59.728610       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get http://<IP_ADDRESS>:8080/api/v1/endpoints?resourceVersion=0: dial tcp <IP_ADDRESS>:8080: getsockopt: connection refused
E0617 03:30:00.749474       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get http://<IP_ADDRESS>:8080/api/v1/services?resourceVersion=0: dial tcp <IP_ADDRESS>:8080: getsockopt: connection refused
E0617 03:30:00.749618       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get http://<IP_ADDRESS>:8080/api/v1/endpoints?resourceVersion=0: dial tcp <IP_ADDRESS>:8080: getsockopt: connection refused

kube-proxy restart fix:
I was able to manually fix the issue on the controller and worker nodes with:
proxies=$(docker ps  | grep -i kube-proxy | awk '{ print $1 }') ; for c in $proxies; do docker stop  $c; sync; sleep 1;   docker rm -v $c; sync; sleep 1;  done ;

docker logs --tail=100 -f  $(docker ps -a | grep kube-proxy | grep -v pause | awk '{ print $1 }')
I0617 04:04:31.563985       1 server.go:225] Using iptables Proxier.
W0617 04:04:31.663074       1 proxier.go:298] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0617 04:04:31.663198       1 server.go:249] Tearing down userspace rules.
I0617 04:04:31.811253       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0617 04:04:31.811926       1 conntrack.go:66] Setting conntrack hashsize to 32768
I0617 04:04:31.812072       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0617 04:04:31.812157       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600

After this, testing the kube-proxy connection to localhost:8080 apiserver worked:
$ docker exec -ti  $(docker ps -a | grep kube-proxy | grep -v pause | awk '{ print $1 }') /bin/sh
$ apt-get update && apt-get -y install curl
$ curl -v 'http://<IP_ADDRESS>:8080/api/v1/endpoints?resourceVersion=0'
* Hostname was NOT found in DNS cache
*   Trying <IP_ADDRESS>...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8080 (#0)
> GET /api/v1/endpoints?resourceVersion=0 HTTP/1.1
> User-Agent: curl/7.38.0
> Host: <IP_ADDRESS>:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Sat, 17 Jun 2017 04:02:38 GMT
< Transfer-Encoding: chunked
<
{
  "kind": "EndpointsList",
  "apiVersion": "v1",

[...SNIP...]
[ JSON api response data here ]

I had to repeat these steps for all nodes, and kube-apiserver began responding well again.  I have not stress tested this, but it works for now.

Update
@mumoshu :
Just to loop back in on this one... We have been implementing AWS NACLs as a separate task in our VPC, and it turns out that we were blocking too much at one point.  This is what caused some of our issues with flanneld, kube-apiserver, kube-proxy, kubelet etc...
However, this also brings up an unhandled case in the etcdadm-reconfigure.service which requires intervention to recover from.  It also is a good way to reproduce the issue for testing purposes 😉
I believe after most of this testing that the current cluster state is not tolerant of full network-level failures and at least requires manual intervention to kickstart recovery of etcd.  We also had to restart flanneld, kubelet, and kube-proxy on all nodes.   When everything goes down the etcdadm-reconfigure systemd unit logs that it gets stuck in handling the case here in /opt/bin/etcdadm:700
The logged message was strange in that it said -1 more nodes are required:
Jun 19 18:15:46 ip-10-123-1-183.ec2.internal etcdadm[5444]: /opt/bin/etcdadm: info: member_reconfigure: observing cluster state: quorum=3 healthy=0
Jun 19 18:15:49 ip-10-123-1-183.ec2.internal etcdadm[5444]: /opt/bin/etcdadm: info: member_reconfigure: -1 more nodes are required until the quorum is met
Jun 19 18:15:49 ip-10-123-1-183.ec2.internal etcdadm[5444]: /opt/bin/etcdadm: info: member_set_unit_type: setting etcd unit type to "notify". `systemctl daemon-reload` required afterwards
Jun 19 18:15:49 ip-10-123-1-183.ec2.internal etcdadm[5444]: /opt/bin/etcdadm: info: member_reconfigure: all the nodes are present but cluster is still unhealthy, which means the initial bootstrap is still in progress. keep retrying a while
Jun 19 18:15:49 ip-10-123-1-183.ec2.internal etcdadm[5444]: /opt/bin/etcdadm: info: _systemctl_daemon_reload: running `systemctl daemon-reload` to reload etcd-member.service
Jun 19 18:15:49 ip-10-123-1-183.ec2.internal systemd[1]: Started etcdadm reconfigure runner.

This message was repeatedly looped each time the etcdadm-reconfigure.service restarted.
We were very confused by this, so we added some debug logging to the script to check the values going into the equation remaining_num=$(( quorum - running_num + 1 )):
  else
    # At least N/2+1 members are NOT working

    local running_num
    local remaining_num
    local total_num
    running_num=$(cluster_num_running_nodes)
    total_num=$member_count
    remaining_num=$(( quorum - running_num + 1 ))

### BEGIN ADDED DEBUG LOGGING
    _info "${running_num} nodes are running"
    _info "${quorum} nodes required for quorum"
    _info "${total_num} nodes total"
### END ADDED DEBUG LOGGING
    _info "${remaining_num} more nodes are required until the quorum is met"

    if (( remaining_num >= 2 )); then
      member_set_unit_type simple
    else
      member_set_unit_type notify
    fi

The resulting log output was:
Jun 19 21:32:35 ip-10-123-1-183.ec2.internal etcdadm[25019]: /opt/bin/etcdadm: info: member_reconfigure: observing cluster state: quorum=3 healthy=0
Jun 19 21:32:39 ip-10-123-1-183.ec2.internal etcdadm[25019]: /opt/bin/etcdadm: info: member_reconfigure: 5 nodes are running
Jun 19 21:32:39 ip-10-123-1-183.ec2.internal etcdadm[25019]: /opt/bin/etcdadm: info: member_reconfigure: 3 nodes required for quorum
Jun 19 21:32:39 ip-10-123-1-183.ec2.internal etcdadm[25019]: /opt/bin/etcdadm: info: member_reconfigure: 5 nodes total
Jun 19 21:32:39 ip-10-123-1-183.ec2.internal etcdadm[25019]: /opt/bin/etcdadm: info: member_reconfigure: -1 more nodes are required until the quorum is met

So this means:

Variable
Value

running_num
5

quorum
3

total_num
5

remaining_num
-1

remaining_num = quorum - running_num + 1
remaining_num = 3 - 5 + 1 = -1

this seems to mean that it calls: member_set_unit_type notify
But the script was not able to recover the entire cluster from this state on it's own without intervention.  We ended up restarting everything as I detailed in the 2 previous comments above here and here.
Additionally, we have still been unable to recover a single etcd node which appears to be a separate issue.  This one is restoring from snapshot, but once etcd-member.service starts up, it fails to re-join the cluster and logs many errors.  I may have to create a separate issue for this one.
I hope this extra information is helpful when working on stabilizing this new etcd recovery feature!
At least this appears to be reproducible by:

Start up a cluster with kube-aws
Set up a NACL rule to break the network connection
Undo NACL rule change
Watch as etcd nodes try to recover but are stuck


Amend user agent

Users can now pass app_name/app_version within the user-agent string (helpful for tracing/troubleshooting).

@vyshakhbabji please let me know if you'd rather the user-agent to default to a blank string (without a slash) instead of having the duplicate values.

@jfriedr  Please verify if this can be merged as well  https://github.com/goshippo/shippo-python-client/pull/73


Can't upload images with customized path

I'm using this plugin (really appreciate for this awesome work!!) on Ruby on Rails 4 and I defined upload path in applicaton.js like below. But it doesn't work. My app try to upload via upload.php as default.
//= require jquery
//= require jquery_ujs
//= require jquery/dist/jquery.min.js
//= require medium-editor/dist/js/medium-editor
//= require handlebars/handlebars.runtime.min.js
//= require jquery-sortable/source/js/jquery-sortable-min.js
//= require blueimp-file-upload/js/vendor/jquery.ui.widget.js
//= require blueimp-file-upload/js/jquery.iframe-transport.js
//= require blueimp-file-upload/js/jquery.fileupload.js
//= require medium-editor-insert-plugin/dist/js/medium-editor-insert-plugin.min.js
//= require jquery-ui
//= require tag-it
//= require materialize-sprockets
//= require_tree .

$( document ).ready(function(){
  $('.editable').bind('input propertychange', function() {
    $("#article_" + $(this).attr("data-field-id")).val($(this).html().replace("#{params[:files]}", "url"));
  });
})

$(function () {
  $(".editable").mediumInsert({
    editor: editor,
    addons: {
      images: {
        fileUploadOptions: {
          url: 'upload_image',
          acceptFileTypes: /(.|\/)(gif|jpe?g|png)$/i
        }
      }
    }
  });
});

Is there anything wrong?

This is strange. When I try it, it works correctly. What versions of medium-editor and the plugin do you use?

I use Medium-Editor v2.0.0 and Medium-Editor-Insert-Plugin v1.0.0. It works correctly on development but doesn't work on production in Heroku. So I suspected this problem is caused by asset compile and I checked precompiled files. And js code I wrote above was included in compiled file (application-XXXXXXX.js).

It's caused by version of the plugin. "fileUploadOptions" must be used with 1.6.0 or later. Careless mistake :-(


new rule: missing visibility modifier on fields

All class fields should have a visibility modifier. Default is public, so accidentally forgetting the modifier gives you the worst possible choice.

fixed in 0.0.5


Rework some required-features tests

Don't delete the build dir as this ends up being flaky on Windows.

@bors: r+ p=10

:pushpin: Commit ec72e71 has been approved by alexcrichton

r? @brson
(rust_highfive has picked a reviewer for you, use r? to override)

:hourglass: Testing commit ec72e71 with merge 01f2018...

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 01f2018df96eaf0b462cd6a78297a38b90a6f281 to master...


Lab 6 typo in section "visualising shift-reduce parsing"

"on the buffer" should be replaced with "in the buffer"

Fixed. Thank you. 1 point


Is the PolyglotMap readonly?

obj = {
    param: "new param value",
    paramArray: [11, 22, "33"],
    paramObj: {
        nestedPar1: "par one",
        nestedIntParam: 333
    }
};

Value returned = bindings.getMember("obj");
Map<String, Object> processedObj = returned.as(Map.class);
List<Object> processedArray = returned.getMember("paramArray").as(List.class);
processedObj.put("paramArray", processedArray);
System.out.println("As List: " + processedArray.getClass().getName());
System.out.println("From Map: " + processedObj.get("paramArray").getClass().getName());

As List: com.oracle.truffle.polyglot.PolyglotList
From Map: com.oracle.truffle.polyglot.PolyglotMap

Not sure I understand what is the issue with the code example you provided, but to answer your question; PolyglotMap is not readonly, you can use remove and put just like any other map.

@oubidar-Abderrahim I thought the code is self explained. Ok, first is the ecmascript code I execute from graaljs. Second is the java code I execute after evaling this script. Java code by lines:

Value#getMember to get member obj from the bindings, which is ecmascript object constructed in the script block above.
Converts returned value to the java Map (i. e. PolyglotMap) and stores it in the processedObj
Converts obj.paramArray, assigned in the line 3 of the script above, to the javaList and stores it in the processedArray
Put processedArray into processedObj with the key "paramArray" replacing existing "paramArray" value
Checks the type of the processedArray variable
Checks the type of the processedObj.get("paramArray") variable
Observe types are different
Conclusion: probably line 4 does not work properly

Hi @BbIKTOP,
I don't think there is an actual problem here. Your assignment actually is a no-op, you assign the same value again (with a different wrapper type, but the underlying data does not effectively change). PolyglotMap and PolyglotList are just wrappers for the actual underlying data, so just changing them does not actually change the underlying data structure (at least not in this case).
If you change that code to actually assign a different value, like:
processedObj.put("paramArray", new int[]{3,2,1}));

Step 7 will actually show different data types.
Best,
Christian


Incorrect image shown in carousel for mobile Safari

Opening this issue as a continuation of the work done in https://github.com/Expensify/App/issues/18150.
cc @kidroca
Outstanding state of things for context, taking @kidroca's comment in https://github.com/Expensify/App/issues/18150#issuecomment-1596793427:

From my end, I no longer encounter the issue of the carousel showing the wrong image during normal usage. The issue appears to be exclusive to mobile Safari on the simulator when I'm testing the app.
I've traced this problem back to the modal handling logic in react-native-web. Currently, I am working on developing a standalone bug example to highlight this issue and intend to open a separate ticket upstream.
If there are no longer reports of the carousel bug, I would propose we close this ticket. We can then initiate a new one specifically to track and monitor the mobile Safari-related issue.

I am also able to reproduce this reliably on mobile Safari

@kidroca you're not appearing in the assignment list for me — are you able to self-assign?

I think I need to comment on the issue and now you should be able to assign me

Here's the related react-native-web issue:

https://github.com/necolas/react-native-web/issues/2544

I've found out the problem happens not only on mobile Safari, but on desktop Safari as well
The bug does not occur on Chrome or the Desktop App which is based on Chromium

I'm closely following the recently created issue in react-native-web:

https://github.com/necolas/react-native-web/issues/2544

As of now, there has been no response or priority assigned to this ticket. This lack of feedback creates uncertainty, making it challenging to work on a fix. It would be helpful to know if the mentioned issue is already being addressed in an upcoming update or if there are any plans to modify the modal focus lock logic.


Added permission to execut

Added permission to execut

When will this one be merged? Is atm a showstopper for me, or is there a simple workaround?

@RdeWilde You can use shivammathur/setup-php@v2 action and specify deployer in the tools parameter. Take a look at this.

Thanks for merging @antonmedv , is there a new version release planned?

@lamasfoker thanks for the suggestion. I tried adding it, but I still get the same error. Can you provide an example on how you got it working?


Auto-saving is not working reliably

We need to implement a more reliable auto-save directive on the front-end side.

This has been fixed


Strings

Can you extract all string resources like in Toasts?

Yes, sorry for missing it.

fixed in v2.5.0


Restore alternatives

Check alternatives with Valdo

@valdok is doing it as a part of the core team effort


[REQUEST] Max daily request per user

It would be cool to assign users with X requests a day or so. Main purpose for this would be to limit abuse, and allows me to check HDD space before anyone requests a lot of things

Thanks for the suggestion, and for using Searcharr. This request is unlikely to be implemented, and will not be worked on at this time. Since Searcharr currently does not keep track of who is adding what, the level of effort to implement this request is too high compared to the benefit. If other users comment to support this request, or other functionality is added that makes this request easier to implement, this request can be re-opened for future consideration.


feature/add test case

changes

update demo page
add error case: missing value(before operator, after operator)
add test code: missing value(after operator)
update type
update npm version

Codecov Report

Merging #7 into master will decrease coverage by 0.09%.
The diff coverage is 85.71%.

@@            Coverage Diff            @@
##           master       #7     +/-   ##
=========================================
- Coverage   93.33%   93.24%   -0.1%
=========================================
  Files          28       28
  Lines         660      666      +6
  Branches       92       95      +3
=========================================
+ Hits          616      621      +5
  Misses         26       26
- Partials       18       19      +1

Impacted Files
Coverage Δ

src/token/token.error.ts
100% <100%> (ø)
:arrow_up:

src/metric.parser.ts
100% <100%> (ø)
:arrow_up:

src/token/token.analyzer.ts
95.34% <75%> (-1%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4b2b1a...5e49117. Read the comment docs.

Pull Request Test Coverage Report for Build 153

5 of 5 (100.0%) changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.05%) to 94.719%

Totals

Change from base Build 148:
-0.05%

Covered Lines:
622

Relevant Lines:
648

💛  - Coveralls


Make taxon lineage versions not nullable

Description
This modifies the SQL schema of taxon_lineages to avoid data corruption.
Notes

Today I deleted all existing rows in taxon_lineages that contained NULL in the two columns
This change will take a few minutes to execute during deployment but it is push-safe because the old and new versions of the schema work with prod code.

Tests
rails db:migrate

Any estimate of how long the deploy will take in prod?

Each alter table modify took 3 min locally, and prod has the same data as local, so I estimate it will add 6 min to the deployment.

Looks like Travis still has an error

Should be fixed now.

Each alter table modify took 3 min locally, and prod has the same data as local, so I estimate it will add 6 min to the deployment.

Ok, heads up @davidrissato / @j-x-han for when this goes out


Benchmarks?

Do you have a set of benchmarks to show that it's actually faster?
using BufferedStreams

x = "id".*string.(rand(UInt16,1_000_000_000))

fn(x) = begin
	io = BufferedOutputStream(open("c:/data/bin.bin", "w"))
	write.(Ref(io), x)
	close(io)
end

gn(x) = begin
	io = open("c:/data/bin2.bin", "w")
	write.(Ref(io), x)
	close(io)
end

using BenchmarkTools

@btime fn($x)
@btime gn($x)

this doesn't indicate that it's faster on Julia 1.2

There were benchmarks we did some time ago on actual biological data files. The numbers were presented during a juliacon talk, I might have them somewhere but it wasn't my work or my talk so I might not have kept them. Bear in mind this package currently performs a legacy role in BioJulia, and you should probably be using TranscodingStreams.jl instead, which does what BufferedStreams does, plus a lot more.

I think you may need to profile to find out where exactly the bottlenecks are, if I benchmark using integers instead of strings:
using BenchmarkTools
using BufferedStreams

julia> y = rand(UInt16,1_000_000)
1000000-element Array{UInt16,1}:
 0x53c2
 0xd42d
 0xbc43
 0x7c79
 0x0be6
 0x4e06
 0xd126
 0x7e1f
 0x1e67
 0x9992
 0xdabe
 0xa8c5
 0x3653
 0x3551
 0xe526
 0xc062
 0x2aac
 0x17c1
 0xe99f
 0x69b3
 0x6efe
 0x4f42
 0x3f31
 0x12e2
 0x92ed
      ⋮
 0x4b33
 0xa6c9
 0x64c1
 0x5080
 0x75bf
 0xb779
 0x4f14
 0x020a
 0x7d62
 0x2eb2
 0x3e63
 0xa036
 0xc77f
 0x2bf9
 0xf21c
 0xa734
 0x3d23
 0x3e1f
 0x975f
 0xfbb7
 0x083f
 0x9fb9
 0x2c72
 0xbaee
 0xc697

julia> function fn(x)
           io = BufferedOutputStream(open("/tmp/bin.bin", "w"))
           for xi in x
               write(io, xi)
           end
           close(io)
       end
fn (generic function with 1 method)

julia> @benchmark fn($y)
BenchmarkTools.Trial:
  memory estimate:  15.39 MiB
  allocs estimate:  1000029
  --------------
  minimum time:     12.579 ms (0.00% GC)
  median time:      15.038 ms (11.38% GC)
  mean time:        14.805 ms (8.38% GC)
  maximum time:     27.942 ms (7.10% GC)
  --------------
  samples:          338
  evals/sample:     1

julia> function gn(x)
           io = open("/tmp/bin.bin", "w")
           for xi in x
               write(io, xi)
           end
           close(io)
       end
gn (generic function with 1 method)

julia> @benchmark gn($y)
BenchmarkTools.Trial:
  memory estimate:  15.26 MiB
  allocs estimate:  1000010
  --------------
  minimum time:     17.986 ms (0.00% GC)
  median time:      20.193 ms (8.41% GC)
  mean time:        19.998 ms (6.13% GC)
  maximum time:     29.700 ms (6.71% GC)
  --------------
  samples:          250
  evals/sample:     1

The write methods of buffered output streams are written in terms of a single UInt8 input or an array of UInt8.
julia> z = rand(UInt8,1_000_000)
1000000-element Array{UInt8,1}:
 0x08
 0xf1
 0x76
 0xc6
 0x55
 0x4e
 0xb3
 0x51
 0x43
 0xdc
 0xcf
 0x52
 0xe5
 0xaa
 0x60
 0x58
 0x78
 0x7c
 0xf7
 0xda
 0x1c
 0x32
 0xff
 0x08
 0x85
    ⋮
 0x99
 0x56
 0x1c
 0x24
 0xbb
 0xc2
 0xc7
 0x27
 0x98
 0x55
 0x1e
 0x10
 0x65
 0x74
 0x72
 0xf0
 0x1a
 0x06
 0x00
 0xef
 0x13
 0x80
 0xfd
 0x25
 0xd2

julia> @benchmark fn($z)
BenchmarkTools.Trial:
  memory estimate:  129.20 KiB
  allocs estimate:  21
  --------------
  minimum time:     1.648 ms (0.00% GC)
  median time:      2.152 ms (0.00% GC)
  mean time:        2.267 ms (0.75% GC)
  maximum time:     13.276 ms (0.00% GC)
  --------------
  samples:          2197
  evals/sample:     1

julia> @benchmark gn($z)
BenchmarkTools.Trial:
  memory estimate:  672 bytes
  allocs estimate:  10
  --------------
  minimum time:     5.561 ms (0.00% GC)
  median time:      6.294 ms (0.00% GC)
  mean time:        6.415 ms (0.00% GC)
  maximum time:     14.218 ms (0.00% GC)
  --------------
  samples:          778
  evals/sample:     1

So it maybe you're doing a LOT of work coercing types or doing something else that has little to do with the streams - if the BufferedStream is calling a write method that has to wait for disk less often than your base IO stream, then it's doing it's job. Any other performance oddities need to be chased down through profiling.


Support specifying iOS shared libraries

This relates to #122, but the bulk of the work must be done in npm-plugin-link. A PR in that repo will follow shortly.

cc @grabbou

LGTM


tmp

Change-Id: Icc1c414fe1185d9843aa5ad6a428267227b92825

Code Review comments can be found at https://gerrithub.io/q/Icc1c414fe1185d9843aa5ad6a428267227b92825

Code Review comments can be found at https://gerrithub.io/q/Icc1c414fe1185d9843aa5ad6a428267227b92825


[Improvement] Introducing APIs to show the progress of scheme change.

What would you like to be improved?
As the schema changes in some catalogs, such as Doris, TiDB is asynchronous and it takes a lot of time for operations like adding an index, so I wonder if we need to add an extra API to show the process of the schema change.
With it, users can easily see when the changes have been completed and then proceed with the next steps.
@zhoukangcn @jerryshao
How should we improve?
No response

Agree it. And I think it would be more appropriate to return the status of schema changes along with the table information.


Recompile Opencv with ffmepg

how to modify root CMakeLists.txt and the next

Can we just use the official opencv4android instead of that compiled  from source code by myself?
And just a quick question, can you provide us with the results( .so/.a) since the process of compile seems too hard for me as a new to opencv?


Fix dialtesting websocket host

fix websocket host, using parsed host, set default header host.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.zhengbo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Tweaked the build property page layout

Tweaked the layout of the build property page:

Aligned all labels horizontally with controls, removed 3 "design pixels" prefix that each label had
Aligned "Generate serialization assembly" label vertically with its control
Changed gaps between controls to match standard spaces (6 "design pixels" vertically between related controls)

To do this, I needed to do a few things first:

Added DependentUpon to the RESX so that they group under their controls
Needed to add the ability to design "abstract" controls via AbstractControlTypeDescriptionProvider
Needed to upgrade Form/resx from "2.0" -> "4.0"

Each of these are in their own commit if you're interested in the details (the churn for 3 is quite great).
Before:

After:

Main changes highlighted:

tag @DustinCampbell @srivatsn

tag @dotnet/roslyn-analysis

👍


Self -> &Self implicit conversion for function lookup doesn't work?

I tried playing with rhai 0.18.1, tried adding a rust type and method, but I'm struggling to get them properly matched if &self for self type is used

#[derive(Clone)]
struct TestStruct {
    x: INT,
}

impl TestStruct {
    fn update(&mut self) {
        self.x += 1000;
    }

    fn get(&self) -> INT {
        self.x
    }

    fn new() -> Self {
        TestStruct { x: 1 }
    }
}

fn test() {
    engine.register_type::<TestStruct>();

    engine.register_fn("update", TestStruct::update);
    engine.register_fn("new_ts", TestStruct::new);
    engine.register_fn("get", TestStruct::get);

    let result = engine.eval::<i32>("let x = new_ts(); x.update(); x.get()").unwrap();
}

will result in ErrorFunctionNotFound("get (test::TestStruct)", 1:33)
looks like implicit conversion from test::TestStruct to &mut test::TestStruct works fine, but test::TestStruct -> &test::TestStruct does not.

:( that makes me sad, which rust feature this depends on, is there an RFC that I can track?

https://github.com/rust-lang/rust/issues/31844

Is this resolved (for now)?  If so I'll close this.  Thanks.


ZipSplit Implementation

Do you think adding zipsplit as option a good idea? For Dropbox at least will help a lot to bypass single size limit.
If yes let me know to do it and PR.
Thanks a lot.

Fixed with #43


[Bug] react 16.8 compatibility issues

Before you submit this issue, have you checked the following

[X] Is this really a problem?
[X] I have searched the Github Issues for similar issues, but did not find anything.

Affected packages and versions
0.1.17
Reproduction link
Using the demo example from: https://github.com/awesome-univer/sheets-react-facade-demo
Change the React version to 16.8.6
The issue is that changing the worksheet name is ineffective  after modification, the data still shows the previous name. Additionally, the event monitor for Command: sheet.command.set-worksheet-name Params was not found.
Expected behavior
The worksheet name can be successfully modified.
Actual behavior
The worksheet name can not be successfully modified.
System information
No response

The lowest compatible version of React for univer is 16.9.0, which is defined in the package.json.

The lowest compatible version of React for univer is 16.9.0, which is defined in the package.json.
Are you sure the minimum compatible version is 16.9.0? Even after upgrading to 16.9.0, I'm still encountering the same issue. After setting the worksheet name, the getActiveWorkbook().getSnapshot() data remains unchanged. I even upgraded to 16.14.0, but it still doesn't work. Could there be a bug here? Or is the minimum compatible version actually higher?
Thank you for your response.

The lowest compatible version of React for univer is 16.9.0, which is defined in the package.json.

The lowest compatible version of React for univer is 16.9.0, which is defined in the package.json.
Are you sure the minimum compatible version is 16.9.0? Even after upgrading to 16.9.0, I'm still encountering the same issue. After setting the worksheet name, the getActiveWorkbook().getSnapshot() data remains unchanged. I even upgraded to 16.14.0, but it still doesn't work. Could there be a bug here? Or is the minimum compatible version actually higher?
Thank you for your response.

It seems like this issue is caused by a known bug in redi. Currently, redi does not support React's strict mode. You can temporarily remove React.StrictMode. We will update the demo repository later.
- <React.StrictMode>
  <App />
- </React.StrictMode>

It seems like this issue is caused by a known bug in redi. Currently, redi does not support React's strict mode. You can temporarily remove React.StrictMode. We will update the demo repository later.
- <React.StrictMode>
  <App />
- </React.StrictMode>

It seems that's not the issue. Even after remove React.StrictMode, it still doesn't work in React versions 16.9.0 or 16.14.0.

Here is a runable demo: https://stackblitz.com/edit/github-hnthoz
Please check if this is caused by caching or other factors.

Here is a runable demo: https://stackblitz.com/edit/github-hnthoz
Please check if this is caused by caching or other factors.

Thanks a lot for your patience and help 🙌. We have discovered and fixed two bugs related to compatibility with React versions prior to 18.0.
The fixes will be released with version 0.1.18.


About input data of GCN

hi,
how to concatenate 2d data(16, 2) with perceptual feature?
what's the size of perceptual feature?
what's the size of input data for GCN?

Hi, thanks for your interest in our work.
Perceptual features are pooled from the intermediate layers of the backbone network.
For example, we pool features from the conv_1 to conv_4 layers from ResNet. Therefore, let C_i be the channel size of the conv_i layer, and the pooled feature size will be (C_1 + C_2 + C_3 + C_4). After concatenated with the joint coordinate (x, y), the final feature size will be (C_1 + C_2 + C_3 + C_4 + 2).
Best,
Long

hi, thanks for your reply. so you meant that the input size is 16*(2+c1+... +c5),  and output size is 163? the architecture of SemGCN in your paper shows input size is 162. 发自我的华为手机-------- 原始邮件 --------主题：Re: [garyzhao/SemGCN] About input data of GCN (#4)发件人：Long Zhao 收件人：garyzhao/SemGCN 抄送：Zhongwei Qiu<PHONE_NUMBER>9@163.com,Author Hi, thanks for your interest in our work.
Perceptual features are pooled from the intermediate layers of the backbone network.
For example, we pool features from the conv_1 to conv_4 layers from ResNet. Therefore, let C_i be the channel size of the conv_i layer, and the pooled feature size will be (C_1 + C_2 + C_3 + C_4). After concatenated with the joint coordinate (x, y), the final feature size will be (C_1 + C_2 + C_3 + C_4 + 2).
Best,
Long
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or mute the thread.

Yes.
162 (as shown in Figure 2) is used for Configuration #1 in our paper which is same in this repo.
16(2+c1+... +c5) is used for Configuration #2 in our paper.
Best,
Long

Yes.
16_2 (as shown in Figure 2) is used for Configuration #1 in our paper which is same in this repo. 16_(2+c1+... +c5) is used for Configuration #2 in our paper.
Best,
Long

Hi,
I still have a problem.
The role of SemGCN is to regress 3d joints coordinate from 2d joints coordinate.
Have you tried to replace the SemGCN model by fc layers?
That means regressing joints coordinates directly by fc layers.
which one is better in your opinion?

Hi, that is exactly what Martinez et al. [1] have done.
[1] Martinez et al. A simple yet effective baseline for 3d human pose estimation. ICCV 2017.
We have implemented it in 'main_linear.py', and the results are shown in the table of this repo. We also have a detailed discussion in our main paper. Please check them accordingly.
Best,
Long


[mechanic] Duels

watching videos from OAA i saw issue that there is still issue of "wussy" contestants who prefer to runing around instead of fighting. My sugestion is to make something to force or limit that. i have few ideas how to do that.

make other duel not wait for end of other (it do benefits for both teams giveing them more time to farm so both teams will be want finish duel fastest way possible,in theory)

make time limit for duels and give win to team that was have better stats (more %hp on end of duel, did more dmg in duel etc.)

after some time start dealing damage based on they max hp (like 2% max hp per second)

Duels are an overall problem for us at the moment , so we've really started looking into them after today's stream. Thanks for the suggestions  ^_^

@VoidsKeeper i saw on videos ;) , that gangbangs after duels can be unnoing ;D

We're already solving this problem with planned high ground capture points in the duel areas. When you sit there undisturbed you get random rune effects, then go hamburgers on the runners.


build: upgrade to solana-* 2.0

Had a look at upgrading to 2.0 and the process seems pretty painful, wondering if you guys have had a look at what would be involved? Seems like a bunch of stuff has moved around + some APIs may have disappeared?
Update
I now see: https://github.com/LiteSVM/litesvm/tree/solana-2

I believe this can be closed as the topic of the issue is currently cooking under the #76


3rd edition players kit

i've got 3 of them i'd like to add to my inventory.
Contents
4 pre-constructed decks
Brujah antitribu deck (50 cards; 6 crypt cards & 44 library cards)
Malkavian antitribu deck (50 cards; 6 crypt cards & 44 library cards)
Tremere antitribu deck (50 cards; 6 crypt cards & 44 library cards)
Tzimsice deck (50 cards; 6 crypt cards & 44 library cards)
Brujah antitribu (Bruce):
Crypt:
Alessandro Garcia; Armin Brenner; Father Juan Carlos; Hektor (sketch variant); Smash; Urraca
Library:
Acrobatics (x2); Amusement Park Hunting Ground; Bewitching Oration (x2); Blood Doll (x2); Bribes (x2); Celerity; Consanguineous Boon (x2); Disputed Territory; Flash (x4); Increased Strength (x2); Iron Glare (x3); Kine Resources Contested (x4); Leather Jacket; Nimble Feet (x2); On the Qui Vive (x2); Potence; Sabbat Priest; Secret Horde; Stunt Cycle (x2); Torn Signpost (x2); Undead Strength (x4); Voter Captivation (x2)
Malkavian antitribu (Malcolm):
Crypt:
Apache Jones; Beauregard Krueller; Bloodfeud; Harold Zettler, Pentex Director (sketch variant); Persephone Tar-Anis; White Lily
Library:
Blood Doll (x2); Cloak the Gathering (x4); Confusion (x3); Dementation; Effective Management; Elder Impersonation; Escaped Mental Patient; Eyes of Chaos (x3); Glancing Blow (x2); Institution Hunting Ground; Kindred Spirits (x3); Leather Jacket (x2); Lost in Crowds (x4); Mind Tricks (x2); Obfuscate; On the Qui Vive (x2); Palatial Estate; Precognition (x2); Sawed-Off Shotgun; Swallowed by the Night (x2); Telepathic Counter (x3); Telepathic Misdirection
Tremere antitribu (Trey):
Crypt:
Esoara (sketch variant); Frondator; Janine; Lucubratio; Paul Cordwood; Uta Kovacs
Library:
Apportation (x4); Blood Doll (x2); Blood Rage (x2); Bonding (x4); Dominate; Enhanced Senses (x3); Govern the Unaligned (x3); Hand Intervention; Harass (x2); On the Qui Vive (x2); Redirection (x3); The Spirit's Touch (x4); Strained Vitae Supply; Thaumaturgy; Theft of Vitae (x4); Threats (x3); University Hunting Ground; Walk of Flame (x3)
Tzimisce (Tim):
Crypt:
Count Vladimir Rustovich (sketch variant); Duality; Radu Bistri; Lady Zara Slatikov; Laika; Yuri Kerezenski
Library:
Amaranth; Aura Reading; Auspex; Blood Doll (x2); Body Arsenal (x3); Bonecraft (x3); Canine Horde; Changeling (x3); Drawing Out the Beast (x2); Enhanced Senses (x4); Festivo dello Estinto; Forced Awakening (x2); Forgery (x2); Guard Dogs (x2); Horrid Form (x2); Library Hunting Ground; Plasmic Form (x2); Raven Spy (x2); Revenant; Rumble (x2); Scorpion Sting (x2); The Spirit's Touch (x2); Vicissitude; Wolf Companion

It is only possible if they ("set/precon" labels of this product in card entries) appear in official vekn cardbase, so the correct repo to raise the issue (or even do the PR) is https://github.com/GiottoVerducci/vtescsv


Bug Report:

Reproduction steps
spotAccountTrade.NewOrder("AAVEUSDT", Side.BUY, OrderType.MARKET, quoteOrderQty: 0.01m) returns {"code":-1100,"msg":"Illegal characters found in parameter 'quoteOrderQty'; legal range is '^([0-9]{1,20})(\.[0-9]{1,20})?$'."}
Expected behavior
order is created
Actual Behavior
{"code":-1100,"msg":"Illegal characters found in parameter 'quoteOrderQty'; legal range is '^([0-9]{1,20})(\.[0-9]{1,20})?$'."}
.NET version
.NET Framework 6.0.x
Operating system
Windows
Environment
No response
dotnet --info
No response

any decimal value with fractional part returns this error

change globalization


croc: 10.0.13 -> 10.1.0

Diff: https://github.com/schollz/croc/compare/v10.0.13...v10.1.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

It seems that some checks failed for no good reason. Can they be rerun?

Probably because croc was moved.

Help me plz

He switch my iPhone

He switch my iPhone

This my serial number iPhone F2LX91DCKPH7


[native_toolchain_c] Android 15 16kb elf alignment

Android 15 has the support to require dylib elf sections to be 16kb aligned.
We should probably have a default flag for this in package:native_toolchain_c so that packages using flutter create --template package_ffi will work out of the box for Android 15 devices with 16kb memory pages.
Context:

https://github.com/flutter/flutter/pull/155508
https://developer.android.com/guide/practices/page-sizes#compile-r26-lower

Also possibly related https://github.com/flutter/flutter/issues/125090

Out of curiosity: Does Android's NDK compiler toolchain not automatically use 16 KB by-default? Or is this for handling older Android compiler toolchains that don't have that default yet?

Related: https://github.com/dart-lang/native/issues/1608

Out of curiosity: Does Android's NDK compiler toolchain not automatically use 16 KB by-default? Or is this for handling older Android compiler toolchains that don't have that default yet?

NDK r27 does with this flag set: -DANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=ON

Out of curiosity: Does Android's NDK compiler toolchain not automatically use 16 KB by-default? Or is this for handling older Android compiler toolchains that don't have that default yet?

We're currently still on r26b on the CI here:
https://github.com/dart-lang/native/blob/16918aef9da60646e5ae0ad0307fe0e1a64355b2/.github/workflows/native.yaml#L55
Also flutter_tools is at 26 at the moment:
https://github.com/flutter/flutter/blob/9be22b58c4bfe8b6cb447de13a500db4e1bd8c33/packages/flutter_tools/gradle/src/main/groovy/flutter.groovy#L58-L63
If you invoke 27 or up directly (not via Groovy, Kotlin, or Application.mk) it also requires passing the flag:
https://developer.android.com/guide/practices/page-sizes#other-build-systems

Bumping the version on the CI here:

https://github.com/dart-lang/native/pull/1612

Workaround before we address this:
    final cbuilder = CBuilder.library(
      name: packageName,
      assetName: '${packageName}_bindings_generated.dart',
      sources: [
        'src/$packageName.c',
      ],
      flags: ['-Wl,-z,max-page-size=16384',], // <----
    );


Adding option to add custom SVG elements

Thank you for this.
I needed to add a custom overlay to one of the carts but found no way to add SVG elements to the charts easily without interfering with the tooltips.
After this change you can do something like:
<BarChart {...props}>
	<path style={{ fill: "none", "stroke": "#444", strokeWidth: "1.5px"}} d="M0,0L0,100"></path>
</BarChart>

To add an extra line to the graph for example. Maybe that's something useful for others too...

Cool thanks! I am rewriting something related to that in my spare time to be able to stack graphs :)
Thanks for the contribution!


The source code for version 1.2.1 is missing

It seems that this repository is out of date. Can you please either update this repository or change the NuGet package to point to the correct repository? Thanks in advance.

Hi Mark,
I guess when I pushed the new code (project.json structure) I did some mess merging the commit.
The new code is under the "src" folder: https://github.com/SharpTools/SharpCache/blob/master/src/SharpCache/MemCache.cs
I will delete the old code as soon as I get home.
Thanks for the heads up and if you could provide some feedback about using it I would really appreciate it :)

I missed this, sorry. Thank you for the fast response.

There you go, I removed the old code :)


Uploading files in Flutter Web

New Issue Checklist

[x] I have searched for a similar issue in the project and found none

Issue Info

Info
Value

Platform Name
Flutter Web

Platform Version
12.0

Dio Version
3.0.7

Android Studio / Xcode Version
. Android Studio 3.5

Repro rate
all the time (100%)

Issue Description and Steps
Issue: Flutter Web File Upload
I'm developing a Flutter Web app. I am able to save data (string values) with Dio without issues. Everything works as expected. However, when I'm trying to upload a file I get this error:
Uncaught (in promise) Error: Unsupported operation: MultipartFile is only supported where dart:io is available.
I'm following the official documentation instructions without success. Any workaround this?

Flutter has not fully support the IO operation on web platform. Check this Link for more details.

Alright, I checked that. Is there a workaround this? Would you consider to update the docs to let others know about not being fully compatible with Flutter Web?

Nevermind, the example code in the package page works.

Nevermind, the example code in the package page works.

Can you share the code which works?? Thank you

Nevermind, the example code in the package page works.

Can you share the code which works?? Thank you

Hi @Chethanpoojary17 , I kinda hit the same issue in flutter web without using dio, this one is working in web, guess you can utilize ".fromBytes" with some uint8List aswell in dio
import 'package:http/http.dart' as http;

[..]

var request = http.MultipartRequest('POST', Uri.parse(serviceUrl));
request.files.add(http.MultipartFile.fromBytes('file', excel.bytes, filename: excel.name));
http.StreamedResponse res = await request.send();

If anyone is still wondering how to get it working on both mobile and web (This is using the image_picker's PickedFile as the image variable type) :
 FormData body;
 final bytes = await image.readAsBytes();
 final MultipartFile file = MultipartFile.fromBytes(bytes, filename: "picture");
 MapEntry<String, MultipartFile> imageEntry = MapEntry("image", file);
 body.files.add(imageEntry);

The catch is that the filename is required.

@aumb Thank you so much. The above solution worked.

@aumb  Very Very Thanks to you for your support .