Node becomes NotReady

We are running EKS in Ireland and our nodes are going unhealthy regularly.
It is not possible to SSH to the host, pods are not reachable. We have experienced this with t2.xlarge, t2.small and t3.medium instances.
We could ssh to another node in the cluster and ping the NotReady node, but are not able to ssh it either.
Graphs show the memory goes high at about the same time that the journalctl logs below. The EBS IO also goes high. The exact time is hard to pinpoint. I added logs with interesting 'failures' around the time that we think the node disappeared.
We are using the cluster for running tests, so pods are getting created and destroyed often.
We have not done anything described in https://github.com/awslabs/amazon-eks-ami/issues/51 for log rotation.
Cluster Information:
CNI: Latest daemonset with image:<PHONE_NUMBER>52.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:1.2.1
Region: eu-west-1
LOGS
** Node AMI**
AMI ID amazon-eks-node-v24 (ami-0c7a4976cb6fafd3a)

** File system **
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1,9G     0  1,9G   0% /dev
tmpfs           1,9G     0  1,9G   0% /dev/shm
tmpfs           1,9G  2,3M  1,9G   1% /run
tmpfs           1,9G     0  1,9G   0% /sys/fs/cgroup
/dev/nvme0n1p1   64G   40G   25G  62% /
tmpfs           389M     0  389M   0% /run/user/1000

** kubectl describe node**
Name: ip-<secret>.eu-west-1.compute.internal
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.medium
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=eu-west-1
failure-domain.beta.kubernetes.io/zone=eu-west-1b
kubernetes.io/hostname=ip-<secret>.eu-west-1.compute.internal
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 30 Oct 2018 11:25:48 +0100
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk Unknown Wed, 31 Oct 2018 10:56:53 +0100 Wed, 31 Oct 2018 10:57:35 +0100 NodeStatusUnknown Kubelet stopped posting node status.
MemoryPressure Unknown Wed, 31 Oct 2018 10:56:53 +0100 Wed, 31 Oct 2018 10:57:35 +0100 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Wed, 31 Oct 2018 10:56:53 +0100 Wed, 31 Oct 2018 10:57:35 +0100 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure False Wed, 31 Oct 2018 10:56:53 +0100 Tue, 30 Oct 2018 11:25:46 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready Unknown Wed, 31 Oct 2018 10:56:53 +0100 Wed, 31 Oct 2018 10:57:35 +0100 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: <secret>
Hostname: ip-<secret>..eu-west-1.compute.internal
Capacity:
cpu: 2
ephemeral-storage: 67096556Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3980344Ki
pods: 17
Allocatable:
cpu: 2
ephemeral-storage:<PHONE_NUMBER>8
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3877944Ki
pods: 17
System Info:
Machine ID: asdf
System UUID: asdf
Boot ID: asdf
Kernel Version: 4.14.62-70.117.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://17.6.2
Kubelet Version: v1.10.3
Kube-Proxy Version: v1.10.3
ProviderID: aws:///eu-west-1b/i-<secret>
Non-terminated Pods: (14 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system aws-node-hshhg 10m (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system kube-proxy-fkrb8 100m (5%) 0 (0%) 0 (0%) 0 (0%)
monitoring datadog-datadog-bk5bd 200m (10%) 200m (10%) 256Mi (6%) 256Mi (6%)
monitoring prometheus-node-exporter-4z2dg 0 (0%) 0 (0%) 0 (0%) 0 (0%)
t1 0 (0%) 0 (0%) 0 (0%) 0 (0%)
t2 0 (0%) 0 (0%) 0 (0%) 0 (0%)
t3 0 (0%) 0 (0%) 0 (0%) 0 (0%)
t4 0 (0%) 0 (0%) 0 (0%) 0 (0%)
t5 250m (12%) 250m (12%) 500Mi (13%) 500Mi (13%)
t6 0 (0%) 0 (0%) 0 (0%) 0 (0%)
t7 250m (12%) 250m (12%) 500Mi (13%) 500Mi (13%)
t8 100m (5%) 0 (0%) 256Mi (6%) 0 (0%)
t9 250m (12%) 250m (12%) 500Mi (13%) 500Mi (13%)
t10 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1160m (57%) 950m (47%)
memory 2012Mi (53%) 1756Mi (46%)
Events: <none>

journalctl logs around the time
okt 31 10:01:29 ip-<secret>.eu-west-1.compute.internal kernel: aws-k8s-agent: page allocation stalls for 10404ms, order:0, mode:0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null)
okt 31 10:01:30 ip-<secret>.eu-west-1.compute.internal kernel: aws-k8s-agent cpuset=1ef2c300b3981b045f3f2fcab050f674afead7e7c828362ec2d40ef82bf02441 mems_allowed=0
okt 31 10:01:31 ip-<secret>.eu-west-1.compute.internal kernel: CPU: 1 PID: 6267 Comm: aws-k8s-agent Not tainted 4.14.62-70.117.amzn2.x86_64 #1
okt 31 10:01:34 ip-<secret>.eu-west-1.compute.internal kernel: Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
okt 31 10:01:36 ip-<secret>.eu-west-1.compute.internal kernel: Call Trace:
okt 31 10:01:38 ip-<secret>.eu-west-1.compute.internal kernel: dump_stack+0x5c/0x82
okt 31 10:01:39 ip-<secret>.eu-west-1.compute.internal kernel: warn_alloc+0x114/0x1c0
okt 31 10:01:41 ip-<secret>.eu-west-1.compute.internal kernel: __alloc_pages_slowpath+0x831/0xe00
okt 31 10:01:42 ip-<secret>.eu-west-1.compute.internal kernel: ? get_page_from_freelist+0x371/0xba0
okt 31 10:01:45 ip-<secret>.eu-west-1.compute.internal kernel: __alloc_pages_nodemask+0x227/0x250
okt 31 10:01:46 ip-<secret>.eu-west-1.compute.internal kernel: filemap_fault+0x204/0x5f0
okt 31 10:01:47 ip-<secret>.eu-west-1.compute.internal kernel: __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs]
okt 31 10:01:50 ip-<secret>.eu-west-1.compute.internal kernel: __do_fault+0x20/0x60
okt 31 10:01:52 ip-<secret>.eu-west-1.compute.internal kernel: handle_pte_fault+0x945/0xeb0
okt 31 10:01:55 ip-<secret>.eu-west-1.compute.internal kernel: __handle_mm_fault+0x431/0x540
okt 31 10:01:57 ip-<secret>.eu-west-1.compute.internal kernel: handle_mm_fault+0xaa/0x1e0
okt 31 10:02:00 ip-<secret>.eu-west-1.compute.internal kernel: __do_page_fault+0x23e/0x4c0
okt 31 10:02:02 ip-<secret>.eu-west-1.compute.internal kernel: ? async_page_fault+0x2f/0x50
okt 31 10:02:07 ip-<secret>.eu-west-1.compute.internal kernel: async_page_fault+0x45/0x50
okt 31 10:02:09 ip-<secret>.eu-west-1.compute.internal kernel: RIP: 0001:0x1f
okt 31 10:02:12 ip-<secret>.eu-west-1.compute.internal kernel: RSP: 0000:000000c420170f58 EFLAGS: 4d32dce245d7
okt 31 10:02:15 ip-<secret>.eu-west-1.compute.internal kernel: Mem-Info:
okt 31 10:02:16 ip-<secret>.eu-west-1.compute.internal kernel: active_anon:895836 inactive_anon:8314 isolated_anon:0
active_file:413 inactive_file:596 isolated_file:0
unevictable:0 dirty:1 writeback:0 unstable:0
slab_reclaimable:17241 slab_unreclaimable:26888
mapped:22510 shmem:28069 pagetables:7173 bounce:0
free:21650 free_pcp:12 free_cma:0
okt 31 10:02:17 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 active_anon:3583344kB inactive_anon:33256kB active_file:1652kB inactive_file:2384kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:90040kB dirty:4kB writeback:0kB shmem:112276kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 16384kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
okt 31 10:02:19 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 DMA free:15620kB min:268kB low:332kB high:396kB active_anon:288kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
okt 31 10:02:22 ip-<secret>.eu-west-1.compute.internal kernel: lowmem_reserve[]: 0 2951 3849 3849
okt 31 10:02:24 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 DMA32 free:54972kB min:51600kB low:64500kB high:77400kB active_anon:2799324kB inactive_anon:31696kB active_file:556kB inactive_file:816kB unevictable:0kB writepending:0kB present:3129320kB managed:3044324kB mlocked:0kB kernel_stack:7968kB pagetables:19844kB bounce:0kB free_pcp:148kB local_pcp:0kB free_cma:0kB
okt 31 10:02:27 ip-<secret>.eu-west-1.compute.internal kernel: lowmem_reserve[]: 0 0 898 898
okt 31 10:02:30 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 Normal free:15408kB min:15708kB low:19632kB high:23556kB active_anon:783732kB inactive_anon:1560kB active_file:884kB inactive_file:1392kB unevictable:0kB writepending:4kB present:987136kB managed:920112kB mlocked:0kB kernel_stack:4304kB pagetables:8848kB bounce:0kB free_pcp:420kB local_pcp:0kB free_cma:0kB
okt 31 10:02:32 ip-<secret>.eu-west-1.compute.internal kernel: lowmem_reserve[]: 0 0 0 0
okt 31 10:02:34 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 2*32kB (UM) 3*64kB (UM) 2*128kB (UM) 1*256kB (U) 1*512kB (M) 2*1024kB (UM) 0*2048kB 3*4096kB (ME) = 15620kB
okt 31 10:02:36 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 DMA32: 1659*4kB (UME) 1496*8kB (UME) 1181*16kB (UME) 446*32kB (UME) 54*64kB (UME) 1*128kB (E) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55356kB
okt 31 10:02:37 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 Normal: 334*4kB (UMEH) 351*8kB (UMEH) 431*16kB (UMEH) 93*32kB (UMEH) 4*64kB (H) 2*128kB (H) 4*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15552kB
okt 31 10:02:38 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
okt 31 10:02:38 ip-<secret>.eu-west-1.compute.internal kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
okt 31 10:02:40 ip-<secret>.eu-west-1.compute.internal kernel: 28769 total pagecache pages
okt 31 10:02:40 ip-<secret>.eu-west-1.compute.internal kernel: 0 pages in swap cache
okt 31 10:02:42 ip-<secret>.eu-west-1.compute.internal kernel: Swap cache stats: add 0, delete 0, find 0/0
okt 31 10:02:43 ip-<secret>.eu-west-1.compute.internal kernel: Free swap = 0kB
okt 31 10:02:49 ip-<secret>.eu-west-1.compute.internal systemd-journal[26209]: Permanent journal is using 392.0M (max allowed 4.0G, trying to leave 4.0G free of 38.6G available → current limit 4.0G).
okt 31 10:02:49 ip-<secret>.eu-west-1.compute.internal kernel: Total swap = 0kB
okt 31 10:02:49 ip-<secret>.eu-west-1.compute.internal kernel: 1033112 pages RAM
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: 0 pages HighMem/MovableOnly
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: 38026 pages reserved
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: process-agent: page allocation stalls for 10580ms, order:0, mode:0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null)
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: process-agent cpuset=67b33ad9edc4663ce3e97ac968df4726a9beeff073706349383b1e9eabd93125 mems_allowed=0
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: CPU: 1 PID: 7452 Comm: process-agent Not tainted 4.14.62-70.117.amzn2.x86_64 #1
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: Call Trace:
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: dump_stack+0x5c/0x82
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: warn_alloc+0x114/0x1c0
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: __alloc_pages_slowpath+0x831/0xe00
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: ? get_page_from_freelist+0x371/0xba0
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: __alloc_pages_nodemask+0x227/0x250
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: filemap_fault+0x204/0x5f0
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs]
okt 31 10:02:50 ip-<secret>.eu-west-1.compute.internal kernel: __do_fault+0x20/0x60
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: handle_pte_fault+0x945/0xeb0
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: ? __switch_to_asm+0x34/0x70
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: ? __switch_to_asm+0x40/0x70
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: __handle_mm_fault+0x431/0x540
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: handle_mm_fault+0xaa/0x1e0
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: __do_page_fault+0x23e/0x4c0
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: ? async_page_fault+0x2f/0x50
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: async_page_fault+0x45/0x50
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: RIP: 00f1:0x11
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: RSP: 0234:00007f768c4e5d38 EFLAGS: 00000000
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: Mem-Info:
okt 31 10:02:51 ip-<secret>.eu-west-1.compute.internal kernel: active_anon:895956 inactive_anon:8314 isolated_anon:0
active_file:15 inactive_file:24 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:17237 slab_unreclaimable:26796
mapped:21743 shmem:28069 pagetables:7196 bounce:0
free:21560 free_pcp:682 free_cma:0

plugin logs
2018-10-31T10:50:15Z [INFO] Starting CNI Plugin v1.2.1  ...
2018-10-31T10:50:15Z [INFO] Received CNI del request: ContainerID(56904923f2dfb96db21ddfb6d39f2429d641141f78511d07823bd315feaf4302) Netns() IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=monitoring;K8S_POD_NAME=datadog-datadog-bk5bd;K8S_POD_INFRA_CONTAINER_ID=56904923f2dfb96db21ddfb6d39f2429d641141f78511d07823bd315feaf4302) Path(/opt/aws-cni/bin:/opt/cni/bin) argsStdinData({"cniVersion":"","name":"aws-cni","type":"aws-cni","vethPrefix":"eni"})
2018-10-31T10:50:15Z [ERROR] Error received from DelNetwork grpc call for pod datadog-datadog-bk5bd namespace monitoring container 56904923f2dfb96db21ddfb6d39f2429d641141f78511d07823bd315feaf4302: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:50051: connect: connection refused"

ipamd.log
2018-10-31T10:05:43Z [DEBUG] Found ip addresses [<IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>] on eni 02:af:21:3c:f9:4e
2018-10-31T10:05:44Z [DEBUG] Found eni mac address : 02:b3:1a:eb:c3:5e
2018-10-31T10:05:52Z [DEBUG] Using device number 0 for primary eni: eni-0f37efb5e4ebecf09
2018-10-31T10:05:52Z [DEBUG] Found eni: eni-0f37efb5e4ebecf09, mac 02:b3:1a:eb:c3:5e, device 0
2018-10-31T10:05:55Z [DEBUG] Found cidr <IP_ADDRESS>/24 for eni 02:b3:1a:eb:c3:5e
2018-10-31T10:05:59Z [DEBUG] Found ip addresses [<IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>] on eni 02:b3:1a:eb:c3:5e
2018-10-31T10:05:59Z [DEBUG] Reconcile existing ENI eni-0ce38d7ac411b07ab IP pool
2018-10-31T10:05:59Z [DEBUG] Reconcile and skip primary IP <IP_ADDRESS> on eni eni-0ce38d7ac411b07ab
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0ce38d7ac411b07ab)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0ce38d7ac411b07ab
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0ce38d7ac411b07ab)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0ce38d7ac411b07ab
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0ce38d7ac411b07ab)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0ce38d7ac411b07ab
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0ce38d7ac411b07ab)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0ce38d7ac411b07ab
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0ce38d7ac411b07ab)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0ce38d7ac411b07ab
2018-10-31T10:06:00Z [DEBUG] Reconcile existing ENI eni-0f1db76fd54b2e3f5 IP pool
2018-10-31T10:06:00Z [DEBUG] Reconcile and skip primary IP <IP_ADDRESS> on eni eni-0f1db76fd54b2e3f5
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f1db76fd54b2e3f5)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f1db76fd54b2e3f5
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f1db76fd54b2e3f5)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f1db76fd54b2e3f5
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f1db76fd54b2e3f5)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f1db76fd54b2e3f5
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f1db76fd54b2e3f5)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f1db76fd54b2e3f5
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f1db76fd54b2e3f5)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f1db76fd54b2e3f5
2018-10-31T10:06:00Z [DEBUG] Reconcile existing ENI eni-0f37efb5e4ebecf09 IP pool
2018-10-31T10:06:00Z [DEBUG] Reconcile and skip primary IP <IP_ADDRESS> on eni eni-0f37efb5e4ebecf09
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f37efb5e4ebecf09)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f37efb5e4ebecf09
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f37efb5e4ebecf09)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f37efb5e4ebecf09
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f37efb5e4ebecf09)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f37efb5e4ebecf09
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f37efb5e4ebecf09)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f37efb5e4ebecf09
2018-10-31T10:06:00Z [DEBUG] Adding ENI(eni-0f37efb5e4ebecf09)'s IPv4 address <IP_ADDRESS> to datastore
2018-10-31T10:06:00Z [DEBUG] IP Address Pool stats: total: 15, assigned: 12
2018-10-31T10:06:00Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0f37efb5e4ebecf09
2018-10-31T10:06:00Z [DEBUG] Successfully Reconciled ENI/IP pool
2018-10-31T10:06:06Z [DEBUG] IP pool stats: total = 15, used = 12, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5
2018-10-31T10:06:06Z [DEBUG] Start increasing IP Pool size
2018-10-31T10:06:06Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3
2018-10-31T10:06:11Z [DEBUG] IP pool stats: total = 15, used = 12, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5
2018-10-31T10:06:12Z [DEBUG] Start increasing IP Pool size
2018-10-31T10:06:13Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3
2018-10-31T10:06:17Z [INFO]  Pods deleted on my node: t1
2018-10-31T10:06:18Z [DEBUG] IP pool stats: total = 15, used = 12, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5
2018-10-31T10:06:19Z [DEBUG] Start increasing IP Pool size
2018-10-31T10:06:19Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3
2018-10-31T10:06:25Z [DEBUG] IP pool stats: total = 15, used = 12, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5
2018-10-31T10:06:25Z [DEBUG] Start increasing IP Pool size
2018-10-31T10:06:26Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3
2018-10-31T10:06:33Z [DEBUG] IP pool stats: total = 15, used = 12, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5
2018-10-31T10:06:34Z [DEBUG] Start increasing IP Pool size
2018-10-31T10:06:35Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3
2018-10-31T10:06:35Z [DEBUG] Reconciling ENI/IP pool info...
2018-10-31T10:07:19Z [INFO]  Pods deleted on my node: t2
2018-10-31T10:09:29Z [INFO]  Pods deleted on my node: t3
2018-10-31T10:14:30Z [ERROR] Failed to retrieve interfaces data from instance metadata RequestError: send request failed
caused by: Get http://<IP_ADDRESS>/latest/meta-data/network/interfaces/macs/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018-10-31T10:14:37Z [ERROR] ip pool reconcile: Failed to get attached eni infoget attached enis: failed to retrieve interfaces data: RequestError: send request failed
caused by: Get http://<IP_ADDRESS>/latest/meta-data/network/interfaces/macs/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018-10-31T10:14:47Z [DEBUG] IP pool stats: total = 15, used = 12, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5
2018-10-31T10:14:49Z [DEBUG] Start increasing IP Pool size
2018-10-31T10:14:51Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3
2018-10-31T10:14:52Z [DEBUG] Reconciling ENI/IP pool info...
2018-10-31T10:14:59Z [INFO]  Pods deleted on my node: t4
2018-10-31T10:15:43Z [DEBUG] Total number of interfaces found: 3
2018-10-31T10:15:44Z [DEBUG] Found eni mac address : 02:63:a9:60:fc:42

Any ideas on this?

All those weird log messages about memory in your journalctl look a lot like OOM messages (but don't specifically mention OOM being started). And the Call Trace line looks like a Kernel bug.
One does free -m say on these nodes?

The issues happening again, here are the logs during the node status becoming NotReady
Journalctl
Oct 30 13:19:02 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: proc-sys-fs-binfmt_misc.automount's automount point already active? Oct 30 13:19:10 ip-10-0-1-121.eu-west-1.compute.internal dhclient[2676]: XMT: Solicit on eth0, interval 108540ms. Oct 30 13:19:16 ip-10-0-1-121.eu-west-1.compute.internal kubelet[3629]: E1030 13:19:16.255590    3629 kubelet_node_status.go:377] Error updating node status, will retry: error getting node "ip-10-0-1-121.eu-west-1.compute.internal": Unauthorized Oct 30 13:19:17 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 25993 (agent) Oct 30 13:19:17 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: proc-sys-fs-binfmt_misc.automount's automount point already active? Oct 30 13:19:17 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 25993 (agent) Oct 30 13:20:47 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 28579 (agent) Oct 30 13:20:47 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: proc-sys-fs-binfmt_misc.automount's automount point already active? Oct 30 13:20:47 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 28579 (agent) Oct 30 13:20:47 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: proc-sys-fs-binfmt_misc.automount's automount point already active? Oct 30 13:20:59 ip-10-0-1-121.eu-west-1.compute.internal dhclient[2676]: XMT: Solicit on eth0, interval 130410ms. Oct 30 13:21:02 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 29006 (agent) Oct 30 13:21:02 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: proc-sys-fs-binfmt_misc.automount's automount point already active? Oct 30 13:21:02 ip-10-0-1-121.eu-west-1.compute.internal systemd[1]: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 29006 (agent)
ipamd.log
2018-11-09T12:35:08Z [DEBUG] Adding ENI(eni-0c6011f1b1a05f13b)'s IPv4 address <IP_ADDRESS> to datastore 2018-11-09T12:35:08Z [DEBUG] IP Address Pool stats: total: 15, assigned: 1 2018-11-09T12:35:08Z [DEBUG] Reconciled IP <IP_ADDRESS> on eni eni-0c6011f1b1a05f13b 2018-11-09T12:35:08Z [DEBUG] Successfully Reconciled ENI/IP pool 2018-11-09T12:35:11Z [INFO]  Pods deleted on my node: t-feature-upate-local-tags/t-feature-upate-local-tags-cypress-2115-xmmkr 2018-11-09T12:35:12Z [INFO]  Pods deleted on my node: t-feature-upate-local-tags/t-feature-upate-local-tags-mobile-2115-75jr5 2018-11-09T12:35:12Z [INFO]  Pods deleted on my node: t-feature-upate-local-tags/t-feature-upate-local-tags-nose-functional-fast-2115-wv5gt 2018-11-09T12:35:13Z [INFO]  Pods deleted on my node: t-feature-upate-local-tags/t-feature-upate-local-tags-nose-functional-slow-2115-rztn4 2018-11-09T12:35:13Z [DEBUG] IP pool stats: total = 15, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:13Z [DEBUG] IP pool stats: total = 15, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:13Z [DEBUG] No ENI can be deleted at this time 2018-11-09T12:35:13Z [ERROR] Failed to decrease pool free ENI: no ENI can be deleted at this time 2018-11-09T12:35:13Z [INFO]  Pods deleted on my node: t-feature-upate-local-tags/t-feature-upate-local-tags-nose-unit-2115-npbdg 2018-11-09T12:35:14Z [INFO]  Pods deleted on my node: t-feature-upate-local-tags/t-feature-upate-local-tags-pytest-2115-2wjj9 2018-11-09T12:35:15Z [INFO]  Add/Update for Pod prometheus-server-5997957687-wlc4w on my node, namespace = monitoring, IP = <IP_ADDRESS> 2018-11-09T12:35:15Z [INFO]  Pods deleted on my node: monitoring/prometheus-server-5997957687-wlc4w 2018-11-09T12:35:18Z [DEBUG] IP pool stats: total = 15, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:18Z [DEBUG] IP pool stats: total = 15, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:18Z [DEBUG] No ENI can be deleted at this time 2018-11-09T12:35:18Z [ERROR] Failed to decrease pool free ENI: no ENI can be deleted at this time 2018-11-09T12:35:23Z [DEBUG] IP pool stats: total = 15, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:23Z [DEBUG] IP pool stats: total = 15, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:23Z [DEBUG] FreeENI: found a deletable ENI eni-0abc9a4f1747dbc0e 2018-11-09T12:35:23Z [INFO] FreeENI eni-0abc9a4f1747dbc0e: IP address pool stats: free 5 addresses, total: 10, assigned: 1 2018-11-09T12:35:23Z [DEBUG] Start freeing eni eni-0abc9a4f1747dbc0e 2018-11-09T12:35:23Z [INFO] Trying to free eni: eni-0abc9a4f1747dbc0e 2018-11-09T12:35:23Z [DEBUG] Found eni eni-0abc9a4f1747dbc0e attachment id: eni-attach-0e57c7548fe9c7da3  2018-11-09T12:35:23Z [DEBUG] Trying to delete eni: eni-0abc9a4f1747dbc0e 2018-11-09T12:35:24Z [DEBUG] Not able to delete eni yet (attempt 1/20): InvalidParameterValue: Network interface 'eni-0abc9a4f1747dbc0e' is currently in use. 	status code: 400, request id: 843583a7-3416-4d7b-b5fd-1393421a5f04  2018-11-09T12:35:29Z [DEBUG] Not able to delete eni yet (attempt 2/20): InvalidParameterValue: Network interface 'eni-0abc9a4f1747dbc0e' is currently in use. 	status code: 400, request id: 1501e10c-2026-4377-8d66-14630c3da533  2018-11-09T12:35:34Z [DEBUG] Not able to delete eni yet (attempt 3/20): InvalidParameterValue: Network interface 'eni-0abc9a4f1747dbc0e' is currently in use. 	status code: 400, request id: 009a38d0-2824-4bb2-b707-6a31cece470f  2018-11-09T12:35:39Z [INFO] Successfully deleted eni: eni-0abc9a4f1747dbc0e 2018-11-09T12:35:39Z [INFO] Successfully freed eni: eni-0abc9a4f1747dbc0e 2018-11-09T12:35:39Z [DEBUG] Successfully decreased IP Pool 2018-11-09T12:35:39Z [DEBUG] IP pool stats: total = 10, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:44Z [DEBUG] IP pool stats: total = 10, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:44Z [DEBUG] IP pool stats: total = 10, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:49Z [DEBUG] IP pool stats: total = 10, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5 2018-11-09T12:35:49Z [DEBUG] IP pool stats: total = 10, used = 1, c.currentMaxAddrsPerENI = 5, c.maxAddrsPerENI = 5

So we are getting no help from AWS via support and here.
Issues:

My guess is that we have an old version of docker, older kernel. Seeing a kernel bug above.
The image does not have log rotation configured for docker by default, causing nodes to fill up with data.
The image has a default ulimit that means that when we use larger instance we run out of file descriptors.

Thus the AMI is not ready for production use! We will have to fork it like the 90 other people who are doing this.
Probably use: Lumslabs or AdvMicrogrid as a base.
Very poor support from AWS :( They told us that the image is setup for default workloads. We are not doing anything special... and we loose two nodes a day.

@agcooke thanks for this report, and sorry for the poor support :( We'll take a look into this

@agcooke Can you email me your case ID? I'll see what I can track down on my end<EMAIL_ADDRESS>

Thanks @micahhausler , I will reach out to you in email and when we figure out the solution let people know.
One thing I did not mention above is that we are using 64GB GP2 EBS volumes on the node.
It's very hand wavy, but it seems that the node that runs prometheus (which uses a GP2 PVC) flaps to NonReady and back every few days. We will investigate using IOPS volumes for instances host and  PVC's to see if it helps resolve the issue. LumoLabs also use an ext4 filesystem and I think a separate volume for docker, that could help with the IO issues.
But I do not believe that IO is the root cause... We should not be seeing kernel level stack traces.

I have seen this on 2 clusters in the last few weeks

We were seeing this frequently. Our current theory is it's related to running out of space on the root filesystem  which caused the kubelet to die silently.

@gflarity Great we will see if that is a cause, we are now running the v25 AMI and it should rotate logs better so the disks should not fill up too much.
We were having these issues on t2.large and t3.medium nodes.
We had a talk with SA's from AWS and they suggested that we try use c5.large or m4.large nodes.

We tried c5.large nodes yesterday, nodes go into NonReady almost immediately.
Busy testing m4.large images, boots OK, will let you know on how it progresses over the next few days.

Summary of the issues we are seeing:

Running out of file descriptors on nodes.

When we get more than +- 20 pods per node we get problems quickly.

IO goes high on Node

Could be connected to Network and/or type of EBS volumes/EC2 Instance types.
Solution could be to use separate volume for docker images.

Running out of IP addresses:

We create and delete pods constantly (mostly jobs, +- 50 pods per hour, not a lot) and then CNI-Plugin runs out of IP's for a node. We are running the latest 1.2.1 Amazon VPC CNI daemonset.

Will keep you up to date with the progress.

@gflarity If you can add some logs to help triage the root cause it would be great!

@agcooke I don't think the rotation fix is in v25, it was just merged two days ago. I might be wrong but when I looked at the dates yesterday v25 was put out like 8 days ago.
When you log into the nodes, is the kubelet silent crashing (ie disappearing). We had this happen a couple days ago and the root fs was full so we think kubelet tried to log before it crashed and couldn't.

What does kubectl get events say? I think you should see a DiskPressure event before the node dies if disk space is the problem?

@max-rocket-internet
(we believe lack of free space is this is the root cause of our previous Not Ready issues, but this particular one we're fairly certain of it)
~/Development/go/src/git.sqcorp.co/cash/squaremason master ./build/sqm kubectl --region=us-west-2 -e staging -- get events 2018-11-16 08:38:04,403245 INFO [1] [region_selectable.go:16] targeting region us-west-2 LAST SEEN   FIRST SEEN   COUNT   NAME                                                            KIND   SUBOBJECT   TYPE      REASON                SOURCE                                                  MESSAGE 15m         1h           14     <redacted>  Node               Warning   FreeDiskSpaceFailed   kubelet, ip-10-<redacted> (combined from similar events): failed to garbage collect required amount of images. Wanted to free<PHONE_NUMBER> bytes, but freed 0 bytes 17s         1h           16      <redacted>   Node               Warning   ImageGCFailed         kubelet, <redacted>   (combined from similar events): failed to garbage collect required amount of images. Wanted to free<PHONE_NUMBER> bytes, but freed 0 bytes

The pattern we see is that there was no logging indicating why kubectl dies... not having space to log is a good excuse for that, and probably the root cause.

@gflarity
I don't get any events when doing kubectl get events
kubectl describe node
ip-192-168-145-119.ec2.internal
Name:               ip-192-168-145-119.ec2.internal
Roles:
Labels:             beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t2.medium
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1b
kubernetes.io/hostname=ip-192-168-145-119.ec2.internal
Annotations:        node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Fri, 02 Nov 2018 22:39:57 -0400
Taints:
Unschedulable:      false
Conditions:
Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason                    Message

OutOfDisk        Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
MemoryPressure   Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
DiskPressure     Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
PIDPressure      False     Tue, 13 Nov 2018 17:26:36 -0500   Fri, 02 Nov 2018 22:39:57 -0400   KubeletHasSufficientPID   kubelet has sufficient PID available
Ready            Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
Addresses:
InternalIP:  <IP_ADDRESS>
Hostname:    ip-192-168-145-119.ec2.internal
Capacity:
cpu:                2
ephemeral-storage:  20959212Ki
hugepages-2Mi:      0
memory:             4040024Ki
pods:               17
Allocatable:
cpu:                2
ephemeral-storage:<PHONE_NUMBER>8
hugepages-2Mi:      0
memory:             3937624Ki
pods:               17
System Info:
Machine ID:                 c1c0616b96eb44c3b7b0d094dcd9536e
System UUID:                EC2CEF1E-2CFC-16F6-F994-9C24FC7754E2
Boot ID:                    38401493-f278-4779-8186-f09cf3b39f5d
Kernel Version:             4.14.62-70.117.amzn2.x86_64
OS Image:                   Amazon Linux 2
Operating System:           linux
Architecture:               amd64
Container Runtime Version:  docker://17.6.2
Kubelet Version:            v1.10.3
Kube-Proxy Version:         v1.10.3
ExternalID:                  i-0857bdc11bf82764b
ProviderID:                  aws:///us-east-1b/i-0857bdc11bf82764b
Non-terminated Pods:         (12 in total)
Namespace                  Name                                      CPU Requests  CPU Limits  Memory Requests  Memory Limits

default                    sleep-8677f78569-6gvt5                    0 (0%)        0 (0%)      0 (0%)           0 (0%)
istio-system               istio-citadel-557f97f46d-dqbbd            10m (0%)      0 (0%)      0 (0%)           0 (0%)
istio-system               istio-sidecar-injector-8d65456b6-6bqn7    10m (0%)      0 (0%)      0 (0%)           0 (0%)
istio-system               istio-telemetry-58cf9fbbfb-prwgd          20m (1%)      0 (0%)      0 (0%)           0 (0%)
istio-system               prometheus-7fcd5846db-89lph               10m (0%)      0 (0%)      0 (0%)           0 (0%)
kube-system                aws-node-rkm2l                            10m (0%)      0 (0%)      0 (0%)           0 (0%)
kube-system                kube-dns-64b69465b4-9ldd7                 260m (13%)    0 (0%)      110Mi (2%)       170Mi (4%)
kube-system                kube-proxy-jtlqm                          100m (5%)     0 (0%)      0 (0%)           0 (0%)
kube-system                tiller-deploy-778f674bf5-r7c4b            0 (0%)        0 (0%)      0 (0%)           0 (0%)
teama                      leads-api-75cdc7b454-cjwmt                110m (5%)     1 (50%)     10Mi (0%)        10Mi (0%)
teama                      loans-ui-847df86c67-m27s6                 110m (5%)     1 (50%)     10Mi (0%)        10Mi (0%)
teamb                      sleep-8677f78569-j982c                    10m (0%)      10m (0%)    10Mi (0%)        10Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests  CPU Limits    Memory Requests  Memory Limits

650m (32%)    2010m (100%)  140Mi (3%)       200Mi (5%)
Events:
PS C:\Temp\repo\EKSPOCv3> kubectl get node
NAME                              STATUS     ROLES     AGE       VERSION
ip-192-168-145-119.ec2.internal   NotReady       12d       v1.10.3
ip-192-168-195-239.ec2.internal   NotReady       12d       v1.10.3
ip-192-168-87-144.ec2.internal    NotReady       6d        v1.10.3
PS C:\Temp\repo\EKSPOCv3> kubectl get node
NAME                              STATUS     ROLES     AGE       VERSION
ip-192-168-145-119.ec2.internal   NotReady       12d       v1.10.3
ip-192-168-195-239.ec2.internal   NotReady       12d       v1.10.3
ip-192-168-87-144.ec2.internal    NotReady       6d        v1.10.3
PS C:\Temp\repo\EKSPOCv3> kubectl describe node ip-192-168-145-119.ec2.internal
Name:               ip-192-168-145-119.ec2.internal
Roles:
Labels:             beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t2.medium
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1b
kubernetes.io/hostname=ip-192-168-145-119.ec2.internal
Annotations:        node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Fri, 02 Nov 2018 22:39:57 -0400
Taints:
Unschedulable:      false
Conditions:
Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason                    Message

OutOfDisk        Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
MemoryPressure   Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
DiskPressure     Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
PIDPressure      False     Tue, 13 Nov 2018 17:26:36 -0500   Fri, 02 Nov 2018 22:39:57 -0400   KubeletHasSufficientPID   kubelet has sufficient PID available
Ready            Unknown   Tue, 13 Nov 2018 17:26:36 -0500   Tue, 13 Nov 2018 17:27:20 -0500   NodeStatusUnknown         Kubelet stopped posting node status.
Addresses:
InternalIP:  <IP_ADDRESS>
Hostname:    ip-192-168-145-119.ec2.internal
Capacity:
cpu:                2
ephemeral-storage:  20959212Ki
hugepages-2Mi:      0
memory:             4040024Ki
pods:               17
Allocatable:
cpu:                2
ephemeral-storage:<PHONE_NUMBER>8
hugepages-2Mi:      0
memory:             3937624Ki
pods:               17
System Info:
Machine ID:                 c1c0616b96eb44c3b7b0d094dcd9536e
System UUID:                EC2CEF1E-2CFC-16F6-F994-9C24FC7754E2
Boot ID:                    38401493-f278-4779-8186-f09cf3b39f5d
Kernel Version:             4.14.62-70.117.amzn2.x86_64
OS Image:                   Amazon Linux 2
Operating System:           linux
Architecture:               amd64
Container Runtime Version:  docker://17.6.2
Kubelet Version:            v1.10.3
Kube-Proxy Version:         v1.10.3
ExternalID:                  i-0857bdc11bf82764b
ProviderID:                  aws:///us-east-1b/i-0857bdc11bf82764b
Non-terminated Pods:         (12 in total)
Namespace                  Name                                      CPU Requests  CPU Limits  Memory Requests  Memory Limits

default                    sleep-8677f78569-6gvt5                    0 (0%)        0 (0%)      0 (0%)           0 (0%)
istio-system               istio-citadel-557f97f46d-dqbbd            10m (0%)      0 (0%)      0 (0%)           0 (0%)
istio-system               istio-sidecar-injector-8d65456b6-6bqn7    10m (0%)      0 (0%)      0 (0%)           0 (0%)
istio-system               istio-telemetry-58cf9fbbfb-prwgd          20m (1%)      0 (0%)      0 (0%)           0 (0%)
istio-system               prometheus-7fcd5846db-89lph               10m (0%)      0 (0%)      0 (0%)           0 (0%)
kube-system                aws-node-rkm2l                            10m (0%)      0 (0%)      0 (0%)           0 (0%)
kube-system                kube-dns-64b69465b4-9ldd7                 260m (13%)    0 (0%)      110Mi (2%)       170Mi (4%)
kube-system                kube-proxy-jtlqm                          100m (5%)     0 (0%)      0 (0%)           0 (0%)
kube-system                tiller-deploy-778f674bf5-r7c4b            0 (0%)        0 (0%)      0 (0%)           0 (0%)
teama                      leads-api-75cdc7b454-cjwmt                110m (5%)     1 (50%)     10Mi (0%)        10Mi (0%)
teama                      loans-ui-847df86c67-m27s6                 110m (5%)     1 (50%)     10Mi (0%)        10Mi (0%)
teamb                      sleep-8677f78569-j982c                    10m (0%)      10m (0%)    10Mi (0%)        10Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests  CPU Limits    Memory Requests  Memory Limits

650m (32%)    2010m (100%)  140Mi (3%)       200Mi (5%)
Events:
Unfortunately, this environment does not have a key pair setup for EC2 so the info that I will be able gather will be limited.
Any suggestions on what else to check without without being on the host? I am going to try autoscaling after adding a key pair, but was avoiding touching anything until i hear from aws support

I see it in the the describe though:
OutOfDisk Unknown Tue, 13 Nov 2018 17:26:36 -0500 Tue, 13 Nov 2018 17:27:20 -0500 NodeStatusUnknown Kubelet stopped posting node status.
MemoryPressure Unknown Tue, 13 Nov 2018 17:26:36 -0500 Tue, 13 Nov 2018 17:27:20 -0500 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 13 Nov 2018 17:26:36 -0500 Tue, 13 Nov 2018 17:27:20 -0500 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure False Tue, 13 Nov 2018 17:26:36 -0500 Fri, 02 Nov 2018 22:39:57 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready Unknown Tue, 13 Nov 2018 17:26:36 -0500 Tue, 13 Nov 2018 17:27:20 -0500 NodeStatusUnknown Kubelet stopped posting node status.

@joberdick
/shrug there might be multiple causes of this...
Can you ssh into the node? That's how I first diagnosed our issue.

We have been running m4.large  and are not seeing the issues anymore.
I think the symptoms are hidden by faster network throughput.

running m4.xlarge and now nodes are dying like crazy....... before it was one a week now it's 3 today already ofc no possibility to ssh and Amazon status checks are saying node is ok (sic!)

@gacopl We are seeing similar behaviour on m4.xlarge as well. Our guess is that https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/ might help, but this is not available on Kubernetes 1.10.3 on AWS. Waiting patiently for Kubernetes 1.11 and 1.12 on AWS EKS :/

keep waiting we started around september and we are still on EKS v1 they did not even upgrade us to v2, choosing EKS was worst decision, the worst part is pods do not get resheduled when node NotReady, will have setup special healthchecks for ASG.....

@gacopl is it helpful to add additional healthchecks even if the node is still unstable?

Since the last two days we are also seeing the m4.large nodes dying all the time...
We are resorting to using another AMI, but do not have results for that yet.

We had relative stability and now things are bad again.

@radityasurya what else do you advise since only terminating node gets pod rescheduled and basic ASG checks do not see the node is unreachable
@agcooke which AMI specifically ?

We are using amazon-eks-node-v25 (ami-00c3b2d35bddd4f5c) and I have started working on using https://github.com/lumoslabs/amazon-eks-ami AMI.
If anyone know how to use a CoreOS AMI or an Ubuntu (conjure-up) AMI it would be great...

@agcooke So v25 does not solve it right ? i was about to upgrade from v23

No, just found https://cloud-images.ubuntu.com/docs/aws/eks/

We are working on an AMI release first week of December with the log rotation patch in. That should fix the issues related to disk filling up due to logs.

@eswarbala It's not only logs but also fd limits are too low on bigger instances

I have some launch user-data stuff i need to keep :/ so Ubuntu is not for me

@kinghajj Yes I saw your work already. I tried the lumolabs Amazon Linux one but we saw nodes crashing.
We moved our entire cluster over to your AMI today. Will let you know how it goes. I will see if I can port some of the nice ulimit changes from the LumoLabs AMI to your AMI.
The ubuntu provided AMI  expects you to have used CloudFormation to create your cluster. So at this time it seems too much to get that working.
If we do not make headway we will recreate the cluster using eksctl. It supports the Ubuntu image now as well and also uses CloudFormation.

@agcooke would it not be better to debug the issue further instead of switching between all the AMIs?

for the moment setting evicition hard nodefs and kube and system reserved mem flags seemed to help we'll see

I should probably open another issue for the AMI, but when we started using some default alerts with the AWS AMI v25, alerts fired frequently for this alert:
ALERT ClockSyncBroken
  IF          node_timex_sync_status != 1
  FOR         5m
  LABELS      { severity="warning" }
  ANNOTATIONS {
    summary = "The clock is not being synced.",
    impact = "Random things are about to break for our users",
    detail = "Node: {{$labels.node}}",
  }

I saw that @albertdixon added NTP settings in their fork. I guess that that is to solve the ClockSyncBroken issue. Should probably be synced back into the upstream AMI.
The Ubuntu based AMI does not have that problem.

@agcooke yes but i reserved little less according to some calculator at k8s.io gave system 500M and kubelet 200M good thing was i was able to set those with kubelet-extra-args in user-data without repacking AMI

How are your feelings regarding Ubuntu AMI is it worth changing to considering they have different bootstrapping etc?

@gacopl At the moment I am more positive. The changes that AdvMicrogrid made to the AMI are minimal, and the base images seems to be setup to work better by default (Newer docker, NTP and ulimits).
The biggest negative, is that it is on ubuntu and I am pretty sure the amazon-eks-ami will never change to that as a base image. Thus we will have to make sure it stays up to date with the upstream.
Using packer to build is very easy. See my branch. I just changed the makefile to use docker-compose to run packer like LumoLabs did.

Anyone have operational experience to know what a sane value is? I think the AWS EKS AMI has a nofile limit of 1000, but not sure, that has been too little for us.

For things like this and many questions of similar fashion, IMO it's best to just work out what GKE does and copy them if possible. Their k8s-as-a-service is 2 years ahead of everyone else 🤭

Since the last update, our ubuntu based nodes have worked properly.
We upgraded to 1.11.5 and the nodes do not work well. Also looking at using the latest (built ourselves) default AWS ami again and it looks OK on one of our less used clusters. Will give feedback how it works on our heavy use cluster when we move there.
Also added https://github.com/awslabs/amazon-eks-ami/pull/130 to use chrony instead of ntp to sync the node times properly.

Added some more pull requests to fix other problems we have encountered:

https://github.com/awslabs/amazon-eks-ami/pull/132 - Also seems to work by increasing ulimits for containers.
https://github.com/awslabs/amazon-eks-ami/pull/133 - Bigger change that makes it easier to upgrade CNI and docker.
https://github.com/awslabs/amazon-eks-ami/pull/134 - Adds a loopback CNI interface that seems to make the nodes boot more reliably. (Not too sure about this one)
https://github.com/awslabs/amazon-eks-ami/pull/135 - Cosmetic changes to allow aws region to be overridden and also use docker-compose instead of packer.

Any solution as of now? We are facing the same issue even after using "--kube-reserved=cpu=500m,memory=500Mi,ephemeral-storage=1Gi --system-reserved=cpu=500m,memory=500Mi,ephemeral-storage=1Gi" ..

@rakeshpatri what AMI are you using?

I am using the eks optimised default ami as per the guide in eks .

Are you using the latest one or at least the one that was pushed in early Jan. It has a fix to tell kubelet to restart if it dies from SIGPIPE which happens if journald gets killed by a lack of resources. If you run cat /etc/systemd/system/kubelet.service | grep "RestartForceExitStatus" on one of your worker nodes, you should see RestartForceExitStatus=SIGPIPE as output. If not, then you don't have the fix that should restart kubelet if journald dies.

@jaredeis on behalf of @rakeshpatri  we  tested it i can see RestartForceExitStatus=SIGPIPE as output
what more can we try ??

I guess the only other thing I could suggest is opening a case with AWS to see if they can help determine why kubelet is dying on your nodes.

I have been seeing the same issue. we are on 2 X t2-medium and one node goes "NotReady" sporadically. The only solution I had was to detach and spawn another node. restart does not work sigh any  interim solution for this? @agcooke / others is your fork available somewhere? clearly aws has no time for this one! 😠

Most of the important changes I made are now merged into the latest AMI.
I would make sure that you are using EBS_Optimized ec2 instances and upgrade your docker version by changing the version in the packer file.
My fork is here https://github.com/agcooke/amazon-eks-ami. I have moved onto another project so have not kept it up to date with the aws image.

I have been seeing the same issue. we are on t2-medium and one node goes "NotReady" sporadically. The only solution I had was to detach and spawn another node. restart does not work sigh any interim solution for this? @agcooke / others is your fork available somewhere? clearly aws has no time for this one! 😠

We have setup cluster Autoscaler as well as have set resource limits on all the deployments in kubernetes. Also use HPA for your deployment that is consuming more resource.

@agcooke were the change released? we are currently running amazon-eks-node-1.11-v20190220 and unfortunately, the issue remains

@tckb I've seen it suggested some of these changes will be in the 1.11.8 AMI's which haven't yet been released, but should be quite soon.

Our EKS cluster(1.11) with AMI (ami-0f54a2f7d2e9c88b3) facing the same issue randomly, and it kills my production services many times per day.
I was wondering if I upgraded the EKS cluster to 1.12 and using the latest AMI ami-0923e4b35a30a5f53 could solve this problem. (follow these steps https://docs.aws.amazon.com/eks/latest/userguide/update-stack.html)

Same issue no Server Version: v1.12.6-eks-d69f1b and AMI ami-0abcb9f9190e867ab.

same issue here.
running with eks 1.12 and latest AMIs on us-east-1

It seems to be caused by the "Out of Memory" error on the kubelet host.
After add the BootstrapArguments  to the cloudformation, NotReady state is no longer happening.

Here is my BootstrapArguments:
--kubelet-extra-args "--kube-reserved memory=0.3Gi,ephemeral-storage=1Gi --system-reserved memory=0.2Gi,ephemeral-storage=1Gi --eviction-hard memory.available<200Mi,nodefs.available<10%"

@benjamin658 can you confirm this? I did not see any of such errors in logs

@benjamin658/ others can you confirm this? I did not see any of such errors in logs

Im not 100 percent sure, but after I added the BootstrapArguments, our cluster is working well now.

Having the same issue.
EKS v1.12.6-eks-d69f1b
AMI ami-0abcb9f9190e867ab

@dijeesh did you try the suggestions from @benjamin658

I'm experiencing the same issues. The problems started when I installed gitlab-runner using helm and spawned ~20 jobs in a hour or so.
Nodes running v1.12.7 on AMI ami-0d741ed58ca5b342e
I have weavescope installed in my cluster and when looking at Hosts/Resources I see many containers named svc-0 (which are coming from gitlab). They are docker containers that should have been deleted (and somehow.. they are, because when I search for them on the nodes using the docker CLI they are gone. kubernetes provides no further information as well). That might be a weavescope bug, but if not: this might be a hint on the node NotReady issues

I thought reserving resources for kubelet was default/built-in behavior of current k8s, but sounds like it is optional and EKS doesn’t do it 😢
The reserved resource for kubelets is extremely important where you use overcommitted workloads (collections of spikey workloads) i.e. any time where resource Limits >= Requests or is you don’t specify resource limits. Under node resource exhaustion you want some workloads to be rescheduled, not entire nodes to go down.
If you are using small nodes, failures like this will be more common. Plus you have the low EKS pod limit caused by ENI limitations. I’d suggest reserving some system resources on each node, and use fewer, larger nodes.
https://github.com/weaveworks/eksctl/issues/795

this still happens on EKS 1.13. It started to happen when cluster running under some really high load.

Happening to me as well, looking at kubectl get node <name> -o=yaml I see taints:
spec:
  providerID: aws:///us-east-1a/i-07b8613b0ed988d73
  taints:
  - effect: NoSchedule
    key: node.kubernetes.io/unreachable
    timeAdded: 2019-07-30T07:11:15Z

I think this might be related? https://github.com/weaveworks/eksctl/issues/795

How could I customize the kubelet to add reserved system resources for an existing cluster / nodegroup?
https://github.com/weaveworks/eksctl/blob/master/site/content/usage/11-customizing-the-kubelet.md

We are seeing similar behaviour, what appears to be almost random/possibly coincides with a deployment. A node or two will suddenly appear to be NotReady, resource graphs indicate utilisation is hardly over 50% so oom shouldn't be an issue.
As mentioned by @AmazingTurtle we are also on 4-5 t3.small nodes with around 50 pods so we may be seeing effects of exhausted network addresses despite not seeing these logs.
In line with @montanaflynn the node has the following taints suddenly applied:
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule

Conditions:
  Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason                   Message
  ----             ------    -----------------                 ------------------                ------                   -------
  MemoryPressure   Unknown   Fri, 29 Nov 2019 14:18:03 +0000   Fri, 29 Nov 2019 14:18:48 +0000   NodeStatusUnknown        Kubelet stopped posting node status.
  DiskPressure     Unknown   Fri, 29 Nov 2019 14:18:03 +0000   Fri, 29 Nov 2019 14:18:48 +0000   NodeStatusUnknown        Kubelet stopped posting node status.
  PIDPressure      Unknown   Fri, 29 Nov 2019 14:18:03 +0000   Fri, 29 Nov 2019 14:18:48 +0000   NodeStatusUnknown        Kubelet stopped posting node status.
  Ready            Unknown   Fri, 29 Nov 2019 14:18:03 +0000   Fri, 29 Nov 2019 14:18:48 +0000   NodeStatusUnknown        Kubelet stopped posting node status.
  OutOfDisk        Unknown   Fri, 29 Nov 2019 14:03:02 +0000   Fri, 29 Nov 2019 14:18:48 +0000   NodeStatusNeverUpdated   Kubelet never posted node status.

Kubernetes version
1.13

Platform version
eks.6

I'm going to try increasing node size and adding some resource limits to deployments that may not have them correctly configured.

@JamesDowning
Unless you're considering to run production deployments on azure, you should go with any other service. Even self-hosted would be better that AWS EKS IMHO.
Have you had a look on Fargate? Using the CDK you can bootstrap apps so easily without worrying about infrastructure. It just runs. I gave it a shot a couple of days ago and it's just sexy and works the way I want it --- not like EKS.

Im getting this on a t3.small node
spec:
  providerID: aws:///eu-west-1b/i-0f46d2a2ae85dde92
  taints:
  - effect: NoSchedule
    key: node.kubernetes.io/unreachable
    timeAdded: "2020-05-20T11:36:58Z"
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    timeAdded: "2020-05-20T11:37:03Z"

What is adding these taints and will they ever get removed?

Seeing this on Amazon EKS 1.26.

What was the resolution this issue as this still persists on eks v1.23?

We are facing this issue also on a daily basis. Any resolution for this?

@dimittal this can happen for many reasons, please open a new issue with details of your environment and symptoms.


documented docker for the auth.json use case

Fixes:

add instructions for the docker run command for the multiple config files use case.

:+1:


TypeScript Loader error when used with Create React App

I'm getting an error when trying to use this with CRA, even with the TS compiler.
Failed to compile.
Module parse failed: The keyword 'interface' is reserved (4:0)
File was processed with these loaders:

./node_modules/source-map-loader/dist/cjs.js
You may need an additional loader to handle the result of these loaders.

It's similar to an issue explain here - https://github.com/facebook/create-react-app/issues/5623 - the final comment suggests that the package should be compiled before publishing.
Hope that helps.

I can look into this, however if you are starting a new project, I recommend that you look into Vite or Next, as CRA is deprecated. They stopped updating it years ago.

I'm seeing the same error in a webpack typescript react app. NOT using cra. wondering if this is just a webpack issue in general
ERROR in<EMAIL_ADDRESS>4:0
Module parse failed: The keyword 'interface' is reserved (4:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
|
|
> interface ScrollToHashElementProps {
|   behavior?: ScrollBehavior;
|   inline?: ScrollLogicalPosition;

I'll check this out.

I've updated to version 2.0.3 which now should output a compiled index.js file into the dist folder.
Please let me know if this resolves the issue for you.

@ncoughlin At the moment, there is no type in the latest version of the package that breaks the build.

I also noticed that you have a commit where you added them, but the latest version does not include this commit


VSCode and Remote-Containers fail

Version
0.6.0
What steps are needed to reproduce the behavior?
Microsoft Windows [Version 10.0.19044.1586]
WSL2
Docker version 20.10.14, build a224086

Install VSCode 1.66.2
Install Remote-Containers plugin 0.231.6

DockerFile
FROM mcr.microsoft.com/vscode/devcontainers/base:0-bionic
devcontainer.json
{ 	/"name": "Ubuntu", 	"build": { 		"dockerfile": "Dockerfile", 		"args": { "VARIANT": "bionic" } 	}, 	"settings": {}, }
What you expected?
I want connect to remote container with stevedore, but there is error
What happened?
Log
`[53 ms] Remote-Containers 0.231.6 in VS Code 1.66.2 (dfd34e8260c270da74b5c2d86d61aee4b6d56977).
[53 ms] Start: Resolving Remote
[63 ms] Setting up container for folder or workspace: c:\worspace\project\Firmware\project.code-workspace
[66 ms] Start: Check Docker is running
[66 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[6963 ms] Server API version: 1.41
[6963 ms] Start: Run: docker volume ls -q
[10335 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=c:\worspace\project\Firmware --filter label=vsch.quality=stable
[12965 ms] Start: Run: C:\Program Files\Microsoft VS Code\Code.exe c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js up --container-data-folder .vscode-server/data/Machine --container-system-data-folder /var/vscode-server --workspace-folder c:\worspace\project\Firmware --workspace-mount-consistency cached --id-label vsch.local.folder=c:\worspace\project\Firmware --id-label vsch.quality=stable --log-level debug --config c:\worspace\project\Firmware.devcontainer\devcontainer.json --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true --terminal-columns 215 --terminal-rows 12
[13172 ms] remote-containers 0.231.6.
[13172 ms] Start: Resolving Remote
[13176 ms] Start: Run: git rev-parse --show-cdup
[17458 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=c:\worspace\project\Firmware --filter label=vsch.quality=stable
[20280 ms] Start: Run: docker build -f c:\worspace\project\Firmware.devcontainer\Dockerfile -t vsc-firmware-ddb69664cfc4183dc0361ff9237ce6d0 --build-arg VARIANT=bionic c:\worspace\project\Firmware.devcontainer
Sending build context to Docker daemon  27.65kB
Step 1/2 : ARG VARIANT="bionic"
Step 2/2 : FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}
---> c7f208b9964f
Successfully built c7f208b9964f
Successfully tagged vsc-firmware-ddb69664cfc4183dc0361ff9237ce6d0:latest
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[29357 ms] Start: Run: docker events --format {{json .}} --filter event=start
[29445 ms] Start: Starting container
[29445 ms] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=c:\worspace\project,target=/workspaces/project,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=c:\worspace\project\Firmware -l vsch.quality=stable --entrypoint /bin/sh vsc-firmware-ddb69664cfc4183dc0361ff9237ce6d0 -c echo Container started
docker: Error response from daemon: failed to start service utility VM (createreadwrite): kernel 'C:\Program Files\Linux Containers\kernel' not found.
See 'docker run --help'.
[32162 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=c:\worspace\project\Firmware --filter label=vsch.quality=stable
[34758 ms] Error: Command failed: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=c:\worspace\project,target=/workspaces/project,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=c:\worspace\project\Firmware -l vsch.quality=stable --entrypoint /bin/sh vsc-firmware-ddb69664cfc4183dc0361ff9237ce6d0 -c echo Container started
[34758 ms] trap "exit 0" 15
[34758 ms] exec "$@"
[34758 ms] while sleep 1 & wait $!; do :; done -
[34758 ms]     at bR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:219:986)
[34758 ms]     at Hw (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:219:924)
[34758 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[34759 ms]     at async IR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:2024)
[34759 ms]     at async Xw (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:3221)
[34759 ms]     at async kR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:13925)
[34759 ms]     at async TR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:13650)
[34793 ms] Exit code 1
[34794 ms] Command failed: C:\Program Files\Microsoft VS Code\Code.exe c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js up --container-data-folder .vscode-server/data/Machine --container-system-data-folder /var/vscode-server --workspace-folder c:\worspace\project\Firmware --workspace-mount-consistency cached --id-label vsch.local.folder=c:\worspace\project\Firmware --id-label vsch.quality=stable --log-level debug --config c:\worspace\project\Firmware.devcontainer\devcontainer.json --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true --terminal-columns 215 --terminal-rows 12
[34794 ms] Exit code 1
`
Additional context
No response

docker: Error response from daemon: failed to start service utility VM (createreadwrite): kernel 'C:\Program Files\Linux Containers\kernel' not found.

I see two issues here:

Most likely Stevedore tray app is not running. It should auto-start after system reboot. If it doesn't, try using Stevedore app in Start menu
Because of previous issue, Docker fallbacks to LCOW, which in turn fails to find C:\Program Files\Linux Containers\kernel. I'm not yet sure why it tries to use that path at all (for me, LCOW works even though I don't have that file). There are similar issues (https://github.com/docker/labs/issues/490, for example), but without any clear way to fix. Needs more investigation.

Docker and Stevedore are started and, for example, download containers
I try to check solution, thanks

One more thing. Could you please check what docker context VSCode uses? It is supposed to be desktop-linux.

Solution not helped to me

3)Context is desktop-linux
Still error
`..........
Successfully built 987862fd2765
Successfully tagged vsc-firmware-ddb69664cfc4183dc0361ff9237ce6d0:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double chec
k and reset permissions for sensitive files and directories.
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[2405259 ms] Start: Run: docker events --format {{json .}} --filter event=start
[2405366 ms] Start: Starting container
[2405367 ms] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=c:\worspace\project,target=/workspaces/project,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=c:\worspace\project\Firmware -l vsch.quality=stable --entrypoint /bin/sh vsc-firmware-ddb69664cfc4183dc0361ff9237ce6d0 -c echo Container started
docker: Error response from daemon: invalid mount config for type "bind": invalid mount path: 'c:/worspace/project' mount path must be absolute.
See 'docker run --help'.
[2408898 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=c:\worspace\project\Firmware --filter label=vsch.quality=stable
[2411909 ms] Error: Command failed: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=c:\worspace\project,target=/workspaces/project,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=c:\worspace\project\Firmware -l vsch.quality=stable --entrypoint /bin/sh vsc-firmware-ddb69664cfc4183dc0361ff9237ce6d0 -c echo Container started
[2411909 ms] trap "exit 0" 15
[2411910 ms] exec "$@"
[2411910 ms] while sleep 1 & wait $!; do :; done -
[2411910 ms]     at bR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:219:986)
[2411910 ms]     at Hw (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:219:924)
[2411910 ms]     at runMicrotasks ()
[2411910 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2411910 ms]     at async IR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:2024)
[2411910 ms]     at async Xw (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:3221)
[2411910 ms]     at async kR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:13925)
[2411910 ms]     at async TR (c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js:224:13650)
[2411953 ms] Exit code 1
[2411969 ms] Command failed: C:\Program Files\Microsoft VS Code\Code.exe c:\Users\user.vscode\extensions\ms-vscode-remote.remote-containers-0.231.6\dist\spec-node\devContainersSpecCLI.js up --container-data-folder .vscode-server/data/Machine --container-system-data-folder /var/vscode-server --workspace-folder c:\worspace\project\Firmware --workspace-mount-consistency cached --id-label vsch.local.folder=c:\worspace\project\Firmware --id-label vsch.quality=stable --log-level debug --config c:\worspace\project\Firmware.devcontainer\devcontainer.json --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true --terminal-columns 215 --terminal-rows 12
[2411969 ms] Exit code 1`

I think I am experiencing same error as @SergeiSOficial:
[2022-04-22T21:06:49.515Z] docker: Error response from daemon: invalid mount config for type "bind": invalid mount path: 'c:/my/Projects/ngx_http_jwted_module' mount path must be absolute

Ok I found https://github.com/microsoft/vscode-remote-release/issues/509 and was able to resolve the issue by specifying the workspaceMount parameter explicitly, changing Windows-style path c:/my/Projects/ngx_http_jwted_module to /mnt/c/Projects/ngx_http_jwted_module.
Everything seems to work now 🎉
Maybe good to mention somewhere in the docs, because this didn't happen in Docker Desktop, so I guess many people will hit it and think that stevedore is to blame...

was able to resolve the issue by specifying the workspaceMount parameter explicitly, changing Windows-style path c:/my/Projects/ngx_http_jwted_module to /mnt/c/Projects/ngx_http_jwted_module.

This should be fixed in just-released Stevedore 0.6.1, so Windows-style path should also work. Feel free to comment if issue still persists. Note that it will take some time to publish release (especially on WinGet where it goes through manual moderation process).

Thanks Marat, appreciate the fix 👍 I'll try it out.

Sorry, took me some time to verify the fix, but unfortunately, it still doesn't work in 0.6.1. Same error. I double-checked that Stevedore was updated. docker-wsl-proxy.exe has date 25.04.2022 and I also rebooted the computer just in case 😄
[2022-05-07T17:02:30.902Z] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=c:\my\Projects\ngx_http_jwted_module,target=/workspaces/ngx_http_jwted_module,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=c:\my\Projects\ngx_http_jwted_module -l vsch.quality=stable --entrypoint /bin/sh vsc-ngx_http_jwted_module-1bd9d58c334ea40c37438f82e31fd112 -c echo Container started
[2022-05-07T17:02:31.299Z]
[2022-05-07T17:02:31.332Z] docker: Error response from daemon: invalid mount config for type "bind": invali
d mount path: 'c:/my/Projects/ngx_http_jwted_module' mount path must be absolute

Stevedore is no longer providing support for Linux containers. See why. I recommend switching to Rancher Desktop, Container Desktop or Colima.


QAT support Strongswan

Hi
I have been trying to accelerate the speed of ipsec connection between two vms (QAT VFs) on a DELL node with 3 QAT devices (C6xx).
Is there any default support for QAT in Stongswan ipsec ? Also how can I enable if it's not?
Your insights will be really helpful.

Hi
I do not know if this will answer all your questions but may I refer you to a white paper : -
https://01.org/sites/default/files/downloads//intelipsecaccelerationwhitepaper.pdf

Hi
I have been trying to accelerate the speed of ipsec connection between two vms (QAT VFs) on a DELL node with 3 QAT devices (C6xx).
Is there any default support for QAT in Stongswan ipsec ? Also how can I enable if it's not?
Your insights will be really helpful.

Did you get this working? It's difficult to find any documentation on how to set this up. My QAT driver is installed and device detected, but now how does Strongswan take advantage of the card? Any special configuration?

Hi
I have been trying to accelerate the speed of ipsec connection between two vms (QAT VFs) on a DELL node with 3 QAT devices (C6xx).
Is there any default support for QAT in Stongswan ipsec ? Also how can I enable if it's not?
Your insights will be really helpful.

Did you get this working? It's difficult to find any documentation on how to set this up. My QAT driver is installed and device detected, but now how does Strongswan take advantage of the card? Any special configuration?


[10/14/2022]: SETTING PAGES

Description
Create a sitting page as illustrated in this Design. This issue was derived from https://github.com/Open-Science-Community-Saudi-Arabia/MOOCs/issues/10 and is for dev_team2 branch

[ ] Create a setting page

Which task are you working on?

[ ] Create a setting page

What needs to be done?

[ ] Use React to render the UI for the in-design Template

Who want to work on this issue?
I want to be assign this task

@BatoolMM I can't find this page on the design can I Come up with a design or the designer will create them?

Yes but you need to create a PR in this repo

@okay ma

hi, @ladyami do you have access to the design? I am working on the course details page.


added search bar

FIX ISSUE: add searchbar #43

Size of search bar is too big, make it small. Also when the preloader loads menubar should not be visible


Volume control on the RPi running MM through the integration

Describe the solution you'd like
I have an 'alsa' sound card (usb sound card) in my RPi, and it would be great if I could control the volume from HA.
Describe alternatives you've considered
I'm trying to adjust the volume by doing some commands via SSH, it's rather difficult to get to work.

I'll look into it, but up until now I have just used the API from https://github.com/Jopyth/MMM-Remote-Control, so this would be an addition outside of that as far as I can tell. Probably not trivial, but it would be awesome to take the integration in that direction.

Maybe it wouldn't be that bad if another module lets us control the volume through MagicMirror's notification-system, that can communicate between modules. Maybe something like this https://github.com/Anonym-tsk/MMM-Volume? MMM-Remote-Control supports sending notifications to other modules as far as I know, but it will probably take some time to test out, as I don't have any audio connected to my mirror.
Just a theory, but it might be possible and within scope.

Agreed. Will keep looking into this at some point.


Fixes

Adding build scripts, changing chips.jsx to be generated on build, adding default prop for informationbar to prevent errors

out to date.
Corrected by some others PR.


Option to change Spacewar game with any other ID

Please add an option to change Spacewar game with any other ID.
Spacewar doesn't support gamepads, so when you use it for non-steam games, you can't use gamepads.
If i could've in my life figure out how to cmake anything, i would do it myself by editing source.

Update:
Changing game ID in 2 .cpp files doesn't change issue with gamepads :\

Used game changes from Spacewar to selected one;
Steam games without remote play work and do have gamepads;
Non-Steam games works but don't have gamepads :c

Before last Steam update we didn't use RemotePlayWhatever for Non-Steam games.
Instead we used steam_appid file trick.
But after last Steam update this way still works, but it leaves a permanent window on top of players screen. Which we can't remove.

Update 2:
Using game folder wipe method instead of steam_appid file doesn't change issue with permanent window on top of client screen :c

Yes, it's not just the ID. You can already change it to any valid AppID using the --appid command line argument. However, to make the controllers work as expected for non-Steam games, it is necessary to initialize the SteamAPI under the same PID, which is not possible without messing with the original game/steam executables, unless we provide it with the appropriate steam_appid.txt file and hope that the in-game overlay will do it for us.

Well, seems like they added some checks, so steam_appid.txt and messing with the original game folder don't let clients to get controls.
It looks like if Steam overlay doesn't recognize game window as active anymore in some weird way. Unless it is really the window of Steam game and used AppID. But it doesn't break RemotePlayWhatever,  i suppose because you are making it as m_enabledDesktopStreaming.


Font Options

This issue is to keep track of various fonts I might come across, that I think might be used as a "shortlist", if I ever decide to change the fonts.

Operator Mono, see #4
Monospaced font. Unfortunately, it doesn't seem to be free. I think their big "invention" was the idea of using a cursive for italics.

Rebond Grotesque.
Originally seen on Arne.me. Also seems to be a paid font, but has "fun" g and k, amoung others.

Sacramento
Available on Google Fonts (so free). One of the new that looks like the connected cursive I learned in school. Doesn't join capitals together well. May be useful if I want to "sign" the site somewhere.

I like this font, particularly the italics. But it doesn't seem to have a reasonable font name... --> https://zed.dev/blog/we-have-to-start-over

Averia Libre had the feel of low cost, old school, analogue printing (where the ink would pool slightly around the letters for to cheap paper).
paired with Caveat, which is a "handwriting font" but stays surrounding readable at smaller sizes.
Originally seem in Magick.css.

IM Fell DW Pica (wow, what a name...) is another "old school, analogue printing" font, available on Google Fonts.

c.f. #33 for a discussion of Drop Cap options, but include fonts Goudy Initialen,  Grenze Gotisch, and  Emily's Candy

Monaspace is a "superfamily" of 5 monospaced fonts.


fix: parse for the CipherType of Shadowsocks in simplified config

Because UnmarshalJSONPB method is not used for parse enum, I add a wrapper meassage for workaround.
Alternative names added for consistency with previous.

Codecov Report
Patch coverage: 3.84% and project coverage change: -0.05 :warning:

Comparison is base (62bef88) 39.05% compared to head (49899af) 39.01%.

:exclamation: Current head 49899af differs from pull request most recent head eda30b2. Consider uploading reports for the commit eda30b2 to get more accurate results

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2510      +/-   ##
==========================================
- Coverage   39.05%   39.01%   -0.05%
==========================================
  Files         617      618       +1
  Lines       37036    37071      +35
==========================================
- Hits        14466    14464       -2
- Misses      20978    21016      +38
+ Partials     1592     1591       -1

Impacted Files
Coverage Δ

proxy/shadowsocks/simplified/config.go
2.56% <0.00%> (-0.57%)
:arrow_down:

proxy/shadowsocks/simplified/errors.generated.go
0.00% <0.00%> (ø)

proxy/shadowsocks/simplified/config.pb.go
12.90% <2.63%> (-3.50%)
:arrow_down:

proxy/shadowsocks/config.go
70.89% <33.33%> (+3.51%)
:arrow_up:

... and 2 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Why admin_client_secret?

First, thanks for this great project.
I have a question:
In docs  you metion:
Modify the admin-cli client
Settings: Service Accounts Enabled
Scope: Full Scope Allowed
Service Account Roles: Select all Client Roles available for the account and realm_management
Why do I need to give admin permissions and realm management to my client API?
If I want only to authenticate users (not create or delete), why cannot I just work with the client_secret?
It just looks like I am giving too much power to my API to just authenticate a user o validate a token.

I guess that is only needed if you also want to manage Keycloak resources like users, roles, etc. If you just want to request tokens you don't need those permissions, but this library has lots of other management capabilities.
However, I will further check / verify this and update the documentation to make this clearer, thanks for the question 👍 I would suggest to leave the issue open and link it to the PR until this is done


[feature] concise response type alias

The response types can be quite long, and inconvenient to use.
const x: Promise<ResponseWithBody<200, OkBody> | ResponseWithBody<404, NotFoundBody>  | ResponseWithBody<400, BadRequestBody>> = undefined;
I propose generating some concise type aliases.
type SomeAlias1 = ResponseWithBody<200, OkBody>;
type SomeAlias2 = ResponseWithBody<404, NotFoundBody>;
type SomeAlias3 = ResponseWithBody<400, BadRequestBody>;

type SomeUnionResponseAlias = SomeAlias1 | SomeAlias2 | SomeAlias3;

The alias's should be backwards compatible, optional to use and work as drop in replacements.

Hi Scott,
I really like the idea.
Just to clarify, when you say optional you mean you would like like to be able to control if they are generated or not?

Just to clarify, when you say optional you mean you would like like to be able to control if they are generated or not?

My original intent was "optional to use" the aliases.
Typescript type definitions are only used at compile time, and there is no additional code generated in the output .js file. So I can't think of any drawbacks of always generating these additional types.

Really like this idea as well! Ensuring we export both the verbose and the consise types would make them optional to use as well. Should not cause any bundle size issues either for the consumer, as as you mention these are just compile time types

@scottc - Wanna give this a shot?


API documentation fails to render

api-console crashes on inline maps (type{} syntax):
https://github.com/mulesoft/api-console/issues/235

Fixed by changing it to the "[]" syntax


Storybook Documentation - Component: Radio Group

Description / User Story
With this task we want to make the Radio Group Component Docs page in Storybook look like the template from design.

Acceptance Criteria

[ ] Component Page has the same structure as the design template (in the following order):

[ ]  Headline & Intro Text incl. a list of anchor links to all following sections
[ ]  Docs Area with Example and all Props (see which props should be displayed for this component in the section below)
[ ]  Appearance

[ ] SizeVariant (incl. sizes = SM, MD, LG)
[ ] Direction(incl. horizontal and vertical)

[ ] Content / Settings

[ ]  Legend
[ ] Without Legend

[ ]  States

[ ] Default
[ ] Disabled  (incl. disabled = true)
[ ] Readonly  (incl. readonly = true)

[ ]  Validation

[ ] Required
[ ] hasError  (incl. hasError = true, once submitted)

[ ] Dependencies

[ ]  FormCaptionGroup

A radio group with all captions (hint text and error message)
A radio group with a hint
A radio group with an error (hasError = true) and a hint text only
A radio group with no captions

[ ]  Component Page includes the texts as defined in the section below
[ ]  Component Page text styles (uppercase, lowercase, spacing) correspond the the design template
[ ]  Component Page spacing corresponds to design template
[ ]  Component Page colors correspond to design template

Props
See this table page "RadioGroup" for all props that should be displayed for the component, incl. their description, default values and the type of controls.
If the props in the list are not the same as the component currently uses, just leave a comment in column G in the excel and a follow-up task will be created. Fixing the components is not part of this ticket.

Content / Texts
See this table page "Documentation Texts" for all texts that should be displayed for the component. Filter column A in cell A3 to only show you the texts of this components.
For the texts please change "[component name]" with "Radio Group".
Also all the examples in the documentation below the Props should not use the default values, but they should use descriptive texts for Label, Placeholder and other text fields. Some examples are already given in the template.

Background information

Example of a collapsible Props Section can be seen here

Feedback @bar-tay :

[ ] Headline: Please rename the page to "Radio Group", please also move it into a folder called "Radio Group" together with the the "Radio" Component. In the end it should be similar to how the current Tooltip is structured on develop, where both the Tooltip as well as the Tooltip Bubble are placed inside the Tooltip Folder.
[ ] Props Panel: Default for errorMessageIcon should be "undefined"
[ ] SizeVariant: Change text from "The Radio component..." to "The Radio Group component..."
[ ] Direction: Remove second "." at the end of the text and also remove or comment-out the whole section as long as there is no direction or layout prop that works
[ ] Required: Change text from "...was not filled, before..." to "...was not checked, before..."
[ ] Has Error: Text is wrong, please use the correct text from the Excel.
[ ] Has Error: Please do not include an error message and an error message icon in the example, just show the error state.
[ ] TOC: Following Links are not working or linking to the correct content:

[ ] Appearance (Headline in section below is missing written wrong)
[ ] Direction (same link as in Size Variant) > remove this, if you removed the section

@bar-tay Thanks for the changes. There is only one small thing left:

[ ] Radio and Radio Group are now correctly placed in a folder, but it is named "Radio" and not "Radio Group"

Looks good, once merged the issue can be closed


MultiError: Refactored MultiError for more concise and safe usage.

Fewer lines of code to use combined error safely.
Goland IDE was marking every usage of the old MultiError "potential nil" error
It was easy to forget using Err() when the error was returned, now it's safely assured on compile time.

NOTE: Potentially I would rename the package to merrors. (: In different PR.
Signed-off-by: Bartlomiej Plotka<EMAIL_ADDRESS>

Thanks for quick review round, rdy for next one :hugs:

@bwplotka can you fix the conflict?


ARM support.

can you provide also an ARM builds?

Thanks for reporting. Implemented by https://github.com/vegardit/docker-traefik-logrotate/commit/95555df7faa3e9eec203919657d355b170332048


Reading the delta lake table will occasionally report that the table does not exist[BUG]

When I indirectly read the data in the table, the reading fails occasionally, and the error is that the table does not exist.

But in fact, this table has always existed. When using the delta1.0.0 version, this problem often occurred when using the DeltaTable.isDeltaTable method. Later, this method was avoided from the business level, and the delta lake was upgraded to 1.0.1. However, when reading, the problem of unrecognized table still occurs occasionally.
The data is stored on azure datalake storage gen2, and a connection is established with delta lake through hadoop-azure. I don't know if it is an internal problem of delta lake, resulting in occasional unrecognized tables.
At present, through the spark.read().format("delta").load() method, will there be an error that the table cannot be recognized
the log is :
java.lang.RuntimeException: org.apache.spark.sql.AnalysisException<EMAIL_ADDRESS>is not a Delta table. 	at com.foxit.sensor.Indicator.CompareWorker.run(CompareWorker.java:41) ~[?:?] 	at com.foxit.sensor.runner.Application.start(Application.java:128) ~[?:?] 	at com.foxit.sensor.runner.Application.main(Application.java:38) ~[?:?] 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?] 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?] 	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?] 	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) ~[spark-core_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:951) ~[spark-core_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) ~[spark-core_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) ~[spark-core_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) ~[spark-core_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.deploy.SparkSubmit$anon$2.doSubmit(SparkSubmit.scala:1030) ~[spark-core_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) ~[spark-core_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-core_2.12-3.1.1.jar:3.1.1] Caused by: org.apache.spark.sql.AnalysisException<EMAIL_ADDRESS>is not a Delta table. 	at org.apache.spark.sql.delta.DeltaErrors$.notADeltaTableException(DeltaErrors.scala:260) ~[?:?] 	at org.apache.spark.sql.delta.catalog.DeltaTableV2.toBaseRelation(DeltaTableV2.scala:142) ~[?:?] 	at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:177) ~[?:?] 	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354) ~[spark-sql_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326) ~[spark-sql_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:306) ~[spark-sql_2.12-3.1.1.jar:3.1.1] 	at scala.Option.map(Option.scala:230) ~[scala-library-2.12.10.jar:?] 	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:266) ~[spark-sql_2.12-3.1.1.jar:3.1.1] 	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240) ~[spark-sql_2.12-3.1.1.jar:3.1.1]

Delta Lake version: 1.0.1
Spark version: 3.1.1
Scala version: 2.12.2

Closing this as it's a duplicate of #1168


fixes opencollective/opencollective#758

Coverage remained the same at 75.894% when pulling a5a35b2fe3e7dcab02a3bfbecaf016f2e43f9e22 on fix758 into cfa6657f28945160c1ad3d10e8e50e457104c179 on master.


replication1

we can't stop the replica or the primary currently when the replication setup.

Learn Build status updates of commit 184a66b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/postgresql/flexible-server/concepts-read-replicas.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@sr-msft
Can you review the proposed changes? IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

changed in another PR, please close this one


LoanField + NRIC

Contains a base class called LoanField, and a new data field for Loan called Nric.
Planning to redo this PR, please do not approve.

 I'll leave NRIC for someone else to implement.


Bad credentials

https://github.com/microsoft/azuredatastudio/actions/runs/97569453
Trying to enable it for this repository ended up with bad credentials message.
Do I need to do something special to pass the credentials in?

Unfortunately there's currently a 1 hour limit on build time. If your workflow takes longer than 1 hour then a token expires and the upload of results fails like this. A fix for this is in the pipeline and should be deployed within a few weeks.
If you can reduce your build time to under 1 hour then that should fix things. I note that all of the times is spent on javascript extraction and analysis. Could you use the paths and paths-ignore config fields (see https://help.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#specifying-directories-to-scan) to exclude some parts of the repo that you are not interested in? Perhaps there's some generated code or libraries checked in to the repo that can be excluded?

Hi, sorry it's taken a long time, but the fix for this issue has now been fully deployed. Jobs can take longer than 1 hour and the update to code scanning won't fail anymore.

@robertbrignull I have a job that takes 1d 18h to complete (yes, it's a huge monorepo but we'd like to see the results of an initial scan result).
What is the current token lifetime? Can that be extended?

Judging by this piece of documentation, the default GitHub Actions job token expires after 24 hours.

@stoyanov-x, this is an old issue. Can you please create a new one and someone on the team can help.
If you are having trouble with the default, generated token, you can add a custom token as a repository secret and use that token in the token input of the codeql-action/upload-sarif action. There are other tricks to try if that doesn't work. We can discuss more in a new issue.


Angular 2/4/5/6 cheatsheet

merge please

Thank you so much!! :)

Your cheatsheet is trending on Reddit. :-)
https://www.reddit.com/r/Angular2/comments/8tsvqu/someone_did_a_cheatsheet_about_angular_2_that/


unified duration parser breaks decimals in %commands

%commands used to allow decimals. not anymore.
notes could allow decimals, if I ban dotted notes.

did it ever work???


Add pt-PT translations

Summary
Added PT-PT translations to locales

Hi @openseabrus,
We try to be careful with the languages we include as each of them add weight to the library.
That being said, you can always add your custom locales.
In this case, since the pt-PT locale share most of the strings with pt-BR, you could use pt-BR as a base and just customize the localization. Let us know if this would work for you.
I'm closing the PR as it's not something we're looking to add to the library, but we appreciate your contribution and encourage you to reach us out again in the future.


How to column sort on basis on labels in grouped table

Issue Type (delete the irrelevant ones)

[x] Question

Specs
What version are you using?
2.19.3
What browser?
chrome
Expected Behavior
What did you expect to happen?
I have created a grouped tables without span. So each column of group has its own label. When i click on header it should sort on basis of label
Actual Behavior
What actually happened?
When i click on header. It sorts children of each group
Steps to Reproduce the Problem
Please detail your steps here

Create grouped table without mode ="span" pass label for each column.
2.Enable sorting
3.I can't figure out how to change sorting from sorting children to sorting on the basis of label

Example .
https://jsfiddle.net/nb6fcqs7
If you see in this link, if you click Name. it will sort children of Senior Dev and Junior Dev and what i want to do is I am looking for is Senior Dev and Junior Dev sorted

hey @rajatjain3571 This is not supported on the table given that header rows may or may not match the table columns. Do you have a suggestion of how sorting of header rows should work? Specially if they don't match the columns?

@xaksis As of now When there is grouped table and we enable sorting. it sorts data inside each group but position of group remains same. I was hoping when there is grouped table and we click on header for sorting it takes all the labels of groups and sort group instead of rows inside the group.
Example for what i am looking is (Just see the output , i edited the json to show you output only)
Before clicking header https://jsfiddle.net/nb6fcqs7
After clicking header https://jsfiddle.net/8w9mhqjc/

Hey @xaksis, I'm actually looking for this feature, I was wondering if there's any update on it ? or if I should start working on a it ? :)
On my side, I would sort headers & rows inside the group aswell.
Thanks for taking time


First day of week changes every time you open the modal

When you set weekStart and weekDays options, and open twice the modal, first day label changes every time you open.
Options:

First open:

Second open:

I just make a pull request solving this.

Thank for your PR, has been merged into the latest release.