Multus: error in invoke Delegate add - "sriov", invalid argument

Just started using the multus-cni, sriov-cni, sriov-device-plugin.
Multus has 2 delegates, Calico on default network, sriov on another.
sriov-device-plugin could see the VFs, "intel.com/sriov_net_A": "16"
$ kubectl get node mtx-huawei2-bld05 -o json | jq '.status.allocatable'
{
"cpu": "64",
"ephemeral-storage": "48294789041",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"intel.com/sriov_net_A": "16",
"intel.com/sriov_net_B": "0",
"memory": "196389160Ki",
"pods": "110"
}
multus conf:
cat 05-multus.conf
{
"name": "multus-cni-network",
"type": "multus",
"delegates": [
{
"name": "k8s-pod-network",
"cniVersion": "0.3.0",
"plugins": [
{
"type": "calico",
"log_level": "info",
"datastore_type": "kubernetes",
"nodename": "mtx-huawei2-bld05",
"mtu": 1440,
"ipam": {
"type": "host-local",
"subnet": "usePodCidr"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
}
]
}
],
"kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
}
sriov crd:
cat sriov-nad-4.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: sriov-net-4
namespace: mtx-test10
annotations:
k8s.v1.cni.cncf.io/resourceName: intel.com/sriov_net_A
spec:
config: '{
"type": "sriov",
"name": "sriov-network-4",
"ipam": {
"type": "host-local",
"subnet": "<IP_ADDRESS>/24",
"routes": [{
"dst": "<IP_ADDRESS>/0"
}],
"gateway": "<IP_ADDRESS>"
}
}'
sriov-device-plugin resources:
cat configMap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: sriovdp-config
namespace: kube-system
data:
config.json: |
{
"resourceList": [{
"resourceName": "sriov_net_A",
"selectors": {
"vendors": ["14e4"],
"devices": ["16af"],
"drivers": ["bnx2x"],
"pfNames": ["eno31"]
}
}
]
}
pod yaml:
cat pod-tc1.yaml
apiVersion: v1
kind: Pod
metadata:
name: testpod1
namespace: mtx-test10
annotations:
k8s.v1.cni.cncf.io/networks: sriov-net-4
spec:
containers:

name: appcntr1
image: centos/tools
imagePullPolicy: IfNotPresent
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 300000; done;" ]
resources:
requests:
intel.com/sriov_net_A: '1'
limits:
intel.com/sriov_net_A: '1'
nodeName: mtx-huawei2-bld05

$ kubectl describe pod testpod1 -n mtx-test10
...
...
Warning  FailedCreatePodSandBox  3m28s (x296 over 42m)  kubelet, mtx-huawei2-bld05  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "1bc793ca392c9745f0ce6352f01bd86bb509567308416b09e02ad9a0011ca70c" network for pod "testpod1": NetworkPlugin cni failed to set up pod "testpod1_mtx-test10" network: Multus: Err adding pod to network "sriov-network-4": Multus: error in invoke Delegate add - "sriov": failed to setup netlink device enp59s4f1 "invalid argument", failed to clean up sandbox container "1bc793ca392c9745f0ce6352f01bd86bb509567308416b09e02ad9a0011ca70c" network for pod "testpod1": NetworkPlugin cni failed to teardown pod "testpod1_mtx-test10" network: Multus: error in invoke Delegate del - "sriov": failed to lookup vf device "net1": Link not found]
/var/log/messages reported:
do-change-link[82]: failure changing link: failure 22 (Invalid argument)
Jun  3 15:37:07 mtx-huawei2-bld05 kernel: [444586.545494] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jun  3 15:37:07 mtx-huawei2-bld05 kubelet[82494]: 2019-06-03 15:37:07.824 [INFO][24437] network.go 380: Disabling IPv4 forwarding ContainerID="fa008dddb917c7d21b86f52555b1641dd03d6f08404af0032c2b48a4d8988389" Namespace="mtx-test10" Pod="testpod1" WorkloadEndpoint="mtx--huawei2--bld05-k8s-testpod1-eth0"
Jun  3 15:37:07 mtx-huawei2-bld05 kernel: [444586.569626] IPv6: ADDRCONF(NETDEV_UP): cali38aee219176: link is not ready
Jun  3 15:37:07 mtx-huawei2-bld05 kubelet[82494]: 2019-06-03 15:37:07.848 [INFO][24437] k8s.go 392: Added Mac, interface name, and active container ID to endpoint ContainerID="fa008dddb917c7d21b86f52555b1641dd03d6f08404af0032c2b48a4d8988389" Namespace="mtx-test10" Pod="testpod1" WorkloadEndpoint="mtx--huawei2--bld05-k8s-testpod1-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"mtx--huawei2--bld05-k8s-testpod1-eth0", GenerateName:"", Namespace:"mtx-test10", SelfLink:"", UID:"1ab4bff0-8650-11e9-bd09-14579fa1adef", ResourceVersion:"9279897", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63695198221, loc:(*time.Location)(0x1ede720)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"projectcalico.org/serviceaccount":"default", "projectcalico.org/namespace":"mtx-test10", "projectcalico.org/orchestrator":"k8s"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"mtx-huawei2-bld05", ContainerID:"fa008dddb917c7d21b86f52555b1641dd03d6f08404af0032c2b48a4d8988389", Pod:"testpod1", Endpoint:"eth0", IPNetworks:[]string{"<IP_ADDRESS>/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.mtx-test10", "ksa.mtx-test10.default"}, InterfaceName:"cali38aee219176", MAC:"36:da:3a:16:f9:62", Ports:[]v3.EndpointPort(nil)}}
Jun  3 15:37:07 mtx-huawei2-bld05 kernel: [444586.570825] IPv6: ADDRCONF(NETDEV_CHANGE): cali38aee219176: link becomes ready
Jun  3 15:37:07 mtx-huawei2-bld05 kernel: [444586.571065] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jun  3 15:37:07 mtx-huawei2-bld05 kubelet[82494]: 2019-06-03 15:37:07.850 [INFO][24437] k8s.go 424: Wrote updated endpoint to datastore ContainerID="fa008dddb917c7d21b86f52555b1641dd03d6f08404af0032c2b48a4d8988389" Namespace="mtx-test10" Pod="testpod1" WorkloadEndpoint="mtx--huawei2--bld05-k8s-testpod1-eth0"
Jun  3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]:   [1559601427.8607] device (cali38aee219176): carrier: link connected
Jun  3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]:   [1559601427.8909] manager: (cali38aee219176): new Veth device (/org/freedesktop/NetworkManager/Devices/2925)
Jun  3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]:   [1559601427.8990] device (enp59s4f1): state change: disconnected -> unmanaged (reason 'removed', sys-iface-state: 'removed')
Jun  3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]:   [1559601427.9197] manager: (dev82): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2926)
Jun  3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]:   [1559601427.9321] device (enp59s4f1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jun  3 15:37:07 mtx-huawei2-bld05 kernel: [444586.655546] IPv6: ADDRCONF(NETDEV_UP): enp59s4f1: link is not ready
Jun  3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]:   [1559601427.9372] platform-linux: do-change-link[82]: failure changing link: failure 22 (Invalid argument)
Jun  3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]:   [1559601427.9384] device (enp59s4f1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Jun  3 15:37:07 mtx-huawei2-bld05 kernel: [444586.660476] IPv6: ADDRCONF(NETDEV_UP): enp59s4f1: link is not ready
Jun  3 15:37:07 mtx-huawei2-bld05 kubelet[82494]: 2019-06-03 15:37:07.982 [INFO][24537] calico.go 442: Extracted identifiers ContainerID="fa008dddb917c7d21b86f52555b1641dd03d6f08404af0032c2b48a4d8988389" Node="mtx-huawei2-bld05" Orchestrator="k8s" WorkloadEndpoint="mtx--huawei2--bld05-k8s-testpod1-eth0"
Jun  3 15:37:07 mtx-huawei2-bld05 kubelet[82494]: 2019-06-03 15:37:07.987 [WARNING][24537] workloadendpoint.go 70: Operation Delete is not supported on WorkloadEndpoint type

From the log message, it seems that device enp59s4f1 is disconnected and changing of link state failed:
Jun 3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]: [1559601427.9372] platform-linux: do-change-link[82]: failure changing link: failure 22 (Invalid argument)
Jun 3 15:37:07 mtx-huawei2-bld05 NetworkManager[28813]: [1559601427.9384] device (enp59s4f1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')

Perhaps check the link state?
If that's not the cause, you might want to enable Multus debug log which may contain details of sriov-cni code failure.

Thank you for the followup.
Thought the state change to disconnected was a result of the failed do-change-link? wonder why it failed...
Enabled multus debug log, please take a look
multus.log
Can't figure out much besides the key message:
2019-06-04T16:59:54-07:00 [error] Multus: error in invoke Delegate add - "sriov": failed to setup netlink device enp59s3f5 "invalid argument"

Thank you for the followup.
Thought the state change to disconnected was a result of the failed do-change-link? wonder why it failed...
Enabled multus debug log, please take a look
multus.log
Can't figure out much besides the key message:
2019-06-04T16:59:54-07:00 [error] Multus: error in invoke Delegate add - "sriov": failed to setup netlink device enp59s3f5 "invalid argument"

Which sriov-cni version are you using? From the log message, it looks like not the latest master, but a release branch?
When vf device is passed to sriov-cni, it tries to first set the link state up and then move the interface to pod namespace; I guess the error happened when netlink library is called to set the VF link up. could you please check if you can set the VF(enp59s3f5) up with ip cmd?
Would you please also make sure that PF device is up ? I don't have a bnx2x card, not sure which mode VF is used to follow PF link state, better to check it's not auto and PF is not down.

1.) The image is as defined in sriov-cni-daemonset.yaml, it's 6-month old but seems to be the latest still. Is there a place to get a better one?
docker images|grep sriov-cni
nfvpe/sriov-cni                             latest                   3ceeb1e50924        6 months ago        263MB
docker pull nfvpe/sriov-cni
Using default tag: latest
latest: Pulling from nfvpe/sriov-cni
Digest: sha256:09dc819e76527d0530a90672016f0f798d961e9f847b4aea719b7b7c56aaa8fd
Status: Image is up to date for nfvpe/sriov-cni:latest
2.) ip addr add works on the failed VF. Is it possible to find out what commands were executed from sriov-cni so we could repeat and see which one resulted in "Invalid argument"?
[root@mtx-huawei2-bld05 net.d]# ip addr show dev enp59s3f5
51: enp59s3f5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 1a:11:c0:10:3d:23 brd ff:ff:ff:ff:ff:ff
[root@mtx-huawei2-bld05 net.d]# ip addr add <IP_ADDRESS>/24 dev enp59s3f5
[root@mtx-huawei2-bld05 net.d]# ip addr show dev enp59s3f5
51: enp59s3f5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 1a:11:c0:10:3d:23 brd ff:ff:ff:ff:ff:ff
inet <IP_ADDRESS>/24 scope global enp59s3f5
valid_lft forever preferred_lft forever
3.)  Another observation, after machine reboots, the 16 VFs are all with "NO-CARRIER,BROADCAST,MULTICAST,UP"
[root@mtx-huawei2-bld05 ~]# ip a |grep enp59s3
27: enp59s3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
30: enp59s3f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
33: enp59s3f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
37: enp59s3f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
43: enp59s3f4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
49: enp59s3f5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
57: enp59s3f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
65: enp59s3f7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
[root@mtx-huawei2-bld05 ~]# ip a |grep enp59s4
74: enp59s4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
82: enp59s4f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
90: enp59s4f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
95: enp59s4f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
102: enp59s4f4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
108: enp59s4f5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
116: enp59s4f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
124: enp59s4f7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
After a failed attempt to start a pod, one of the VFs -- probably the chosen one for the pod -- will come back in state:
102: enp59s4f4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
Thanks.  -Jessica

1.) The image is as defined in sriov-cni-daemonset.yaml, it's 6-month old but seems to be the latest still. Is there a place to get a better one?

Yes, it seems an old one, maybe due to the image under nfvpe/sriov-cni is not updated properly when new PR merged. Anyway, you can build the sriov-cni binary(execute make build) from latest code and copy build/sriov to /opt/cni/bin/ directory, then Multus will be able to use the newly built sriov-cni binary. Then we can capture the new log message and see where the error occurs.

docker images|grep sriov-cni
nfvpe/sriov-cni latest 3ceeb1e50924 6 months ago 263MB
docker pull nfvpe/sriov-cni
Using default tag: latest
latest: Pulling from nfvpe/sriov-cni
Digest: sha256:09dc819e76527d0530a90672016f0f798d961e9f847b4aea719b7b7c56aaa8fd
Status: Image is up to date for nfvpe/sriov-cni:latest
2.) ip addr add works on the failed VF. Is it possible to find out what commands were executed from sriov-cni so we could repeat and see which one resulted in "Invalid argument"?

From previous log message, the error happened at here; so we are using github.com/vishvananda/netlink library to set up the link state, the exact function called is LinkSetUp.

[root@mtx-huawei2-bld05 net.d]# ip addr show dev enp59s3f5
51: enp59s3f5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 1a:11:c0:10:3d:23 brd ff:ff:ff:ff:ff:ff
[root@mtx-huawei2-bld05 net.d]# ip addr add <IP_ADDRESS>/24 dev enp59s3f5
[root@mtx-huawei2-bld05 net.d]# ip addr show dev enp59s3f5
51: enp59s3f5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 1a:11:c0:10:3d:23 brd ff:ff:ff:ff:ff:ff
inet <IP_ADDRESS>/24 scope global enp59s3f5
valid_lft forever preferred_lft forever
3.) Regarding link state, didn't play with it, guess auto is default, the PF is an active slave of a bond, should it be set to enable or something?

 Another observation, after machine reboots, the 16 VFs are all with "NO-CARRIER,BROADCAST,MULTICAST,UP"
[root@mtx-huawei2-bld05 ~]# ip a |grep enp59s3
27: enp59s3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
30: enp59s3f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
33: enp59s3f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
37: enp59s3f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
43: enp59s3f4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
49: enp59s3f5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
57: enp59s3f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
65: enp59s3f7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
[root@mtx-huawei2-bld05 ~]# ip a |grep enp59s4
74: enp59s4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
82: enp59s4f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
90: enp59s4f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
95: enp59s4f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
102: enp59s4f4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
108: enp59s4f5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
116: enp59s4f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
124: enp59s4f7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
After a failed attempt to start a pod, one of the VFs -- probably the chosen one for the pod -- will come back in state:
102: enp59s4f4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
Thanks. -Jessica

Built sriov from latest code, the messages are a bit different, but essentially the same result.
The pointers to code helped narrow down the issue, assuming the steps in function SetupVF are equivalent to running ip link commands, I could run ip link set xxx down, then up, on physical NIC, but not on VFs.
Following up with lab manager and redhat on this... will update
[root@mtx-huawei2-bld05 log]# ip link |grep eno51
7: eno51: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set eno51 down
[root@mtx-huawei2-bld05 log]# ip link |grep eno51
7: eno51: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc mq master bond1 state DOWN mode DEFAULT group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set eno51 up
[root@mtx-huawei2-bld05 log]# ip link |grep eno51
7: eno51: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip a |grep enp59s3f1
30: enp59s3f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set enp59s3f1 down
[root@mtx-huawei2-bld05 log]# ip a |grep enp59s3f1
30: enp59s3f1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set enp59s3f1 up
RTNETLINK answers: Invalid argument
Thanks.  -Jessica

Attached multus.log from new code
multus.log
2019-06-06T12:12:18-07:00 [error] Multus: error in invoke Delegate add - "sriov": failed to set up pod interface "net1" from the device "eno31": error setting up interface in container namespace: "error bringing interface up in container ns: "invalid argument""
That's how I went to SetupVF

Built sriov from latest code, the messages are a bit different, but essentially the same result.
The pointers to code helped narrow down the issue, assuming the steps in function SetupVF are equivalent to running ip link commands, I could run ip link set xxx down, then up, on physical NIC, but not on VFs.

Yes, it should be equivalent to running ip cmd, as both are using netlink to configure link state.
The new log message shows that it errors at the same function [LinkSetUp](https://github.com/intel/sriov-cni/blob/master/pkg/sriov/sriov.go#L138), my guess is that the vf driver doesn't support the config option?

Following up with lab manager and redhat on this... will update
[root@mtx-huawei2-bld05 log]# ip link |grep eno51
7: eno51: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set eno51 down
[root@mtx-huawei2-bld05 log]# ip link |grep eno51
7: eno51: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc mq master bond1 state DOWN mode DEFAULT group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set eno51 up
[root@mtx-huawei2-bld05 log]# ip link |grep eno51
7: eno51: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip a |grep enp59s3f1
30: enp59s3f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set enp59s3f1 down
[root@mtx-huawei2-bld05 log]# ip a |grep enp59s3f1
30: enp59s3f1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
[root@mtx-huawei2-bld05 log]# ip link set enp59s3f1 up
RTNETLINK answers: Invalid argument
Thanks. -Jessica

Yes eno51 is a physical nic, ip link commands worked, enp59s3f1 is a VF, failed, will come back.
Thanks.  -Jessica

@hymgg Thank you for reporting this issue. We have recently applied few changes to update the docker image. The original issue described here should be gone now. Feel free to open this issue if you are still experiencing same problem.


Build SDK with musl

This tracker is for issues related to:

Common Front End (CFE) and kernel

Dart VM

dev_compiler

Whether you are using: Void Linux x64 musl

It it somehow possible to build Dart using musl compiler instead of glibc?

Right now we don't provide a way to build Dart with musl compiler.

I've looked into this a bit and the initial blocker is the fact that the build process requires a pre-build dart binary dynamically linked to glibc. Thus you'd need to bootstrap Dart without using pre-built binaries, and so far I've not found instructions or any efforts to do so.

Any update on this?

Well, Alpine Linux has been shipping Dart for quite a bit.
https://pkgs.alpinelinux.org/package/edge/testing/x86_64/dart
Bootstrapped from the Windows version of Dart through Wine...

The wine/dart does not help really if you run docker on Mac M1
(yes I know that I can change the arch of a container)
also, adding wine might remove the size advantage, and it's a super ugly workaround
It would be really nice if dart-SDK would build on alpine. There are quite some other projects that require it and have now real inconvenient situations for users.
One example is static blog site generators that used the old ruby sass and now switch to dart sass. And ups, the container you had does not work anymore. Sory user. That is kind of really bad

I agree, it's really, really bad. But it's a way to get something running. Musl itself isn't really a problem, Dart only needs a few small patches to get it working which should be easy enough to upstream.
The issue is the bootstrapping. Dart requires Dart to compile, but when Dart doesn't run on the platform... well you need to have a workaround. In Alpine's case this is Wine right now, but I rather do a full from-source bootstrap, starting from the very last version that doesn't require itself to compile (which is as far as I can see Dart 2.0.0-dev.8.0) and then going up version by version till you're on the most recent one. It's a better chain of trust but also makes it easier to compile it for new architectures, as Wine on anything non-x86 is not that great.
As I said earlier I looked into this and I have made quite a bit of progress since then, but didn't manage to finish it yet. Currently I have Dart 2.0.0-dev.56.0 running natively on Musl compiled through 3 earlier Dart versions, without requiring Wine or glibc anywhere. I'm stuck here however, the next step would've been Dart 2.0.0-dev-65.0 but it fails for me there at the moment.
It's a pain, and I wish Google cared at least a bit about this. :cry:

We could merge the necessary runtime patches (given that they are small).
As for bootstrapping. I don't think we will provide a full from-source solution, but we could theoretically provide a solution which does not require a prebuilt Dart binary: we simply need to maintain a fixed copy of Dart front-end compiled to Kernel AST, and update it whenever Kernel AST format changes or whenever we update SDK prebuilts. Bootstrapping process can then use this prebuilt CFE to compile the rest of Dart code.
I see that Alpine is quite popular for Docker, so maybe we should consider doing that. /cc @athomas @mit-mit

Using a pre-built anything is not great from a distribution point of view, but it would be way better already than having to bootstrap from a Windows pre-built Dart. So I'd welcome that change yes, thanks.
I believe Alpine is the most popular image for Docker containers, but there are also people happy if Dart (and especially Flutter) would run on Alpine outside of Docker :wink:

I think #50998 can potentially solve the bootstrap problem. When Linux SDK is built without tcmalloc, it works on musl-libc+gcompat.

The Alpine Wiki mentions three ways to run glibc programs

gcompat
flatpack
chroot

ref: https://wiki.alpinelinux.org/wiki/Running_glibc_programs
There are also several user projects, one that might be interesting for you could be
https://hub.docker.com/r/frolvlad/alpine-glibc/
I do not know if anything of that could help you with the bootstrap problem, but I thought it will also not harm to add this info here

I know how to run such things, but the current glibc Dart doesn't run through gcompat. You need a Dart build with #50998 for that, a.k.a. not the binaries Google currently offers for download. Which is why I'm asking if they're going to offer binaries with tcmalloc disabled for download so that can be used to build a properly native Musl binary.

I know it's bad practice to +1 these things, but I thought I might add that a few widely used Jekyll plugins have started upgrading their dependencies (e.g. jekyll-include-cache), which included a new transient dependency on the sass-embedded gem, which requires Dart, and most Jekyll build containers are Alpine, so you might be seeing more activity around this.
The workaround suggested was to pin the version of a dependency, which will get people by for now, but nobody should have to remain committed to this workaround forever for a number of reasons. We could all probably change our deployment pipelines, but I think this would affect enough people to justify the benefit of adding musl support.
Thanks for all of the hard work so far! Looks like you all have made some pretty awesome progress lately! Can't wait for this change!

I started a project to release unofficial Dart SDK for musl: https://github.com/dart-musl/dart
Unofficial 2.18.7 release for musl is available here: https://github.com/dart-musl/dart/releases/tag/2.18.7
Also available as alpine based container: docker pull ghcr.io/dart-musl/dart

The project works as follow:

Bootstrap the previous version (currently 2.17.0) of Dart required to build new Dart on Debian with dart_use_tcmalloc=false.
Patch the new version (currently 2.18.7):

Use correct cross compile target for alpine.
Use alpine provided host clang toolchain.
A few other minor changes...

Replace debian based sysroot with alpine based sysroot.
Replace the bootstrap SDK in source tree with the custom build with no tcmalloc.
Install gcompat on alpine so that the bootstrap SDK works.

Note that this only works on 64-bit systems due to gcompat issue: https://git.adelielinux.org/adelie/gcompat/-/issues/357

Copy 32-bit musl to 64-bit alpine to support cross compile 32-bit.
Compile or cross-compile with dart_use_tcmalloc=false against alpine based sysroots.

The whole build and release process is done by GitHub Actions, and you can see the source code if you're interested .
I'd happy to contribute this to upstream, but without access to Google's internal infrastructure it is impossible to do it properly. The exact build steps and patch sets I used are open source, so if anyone from Google is willing to take it with Google's infrastructure to provide official musl support, please free feel.

Known issues:
The arm build do not work on qemu-user-static, which seems to be a long existing qemu bug where pthread_getattr_np keeps loop over mremap until the address become negative. A previous report of the same issue can be found here: https://www.openwall.com/lists/musl/2017/06/15/9.
There might also be other unknown issues. Please give it a try.

@ntkme could you make PRs with code patches you have mentioned before? I am happy to merge them if they are reasonable.
We also started discussions internally on providing a bootstrap process which does not require a dart binary to be available (by publishing a Kernel binary version of CFE to CIPD and providing GN changes which allow you to use this binary instead of full dart binary to bootstrap). I can't provide concrete timeline on when this becomes available - but this is something we are going to look at.

@mraleph That sounds great. It will take some time to prepare the PRs as what was replace the Debian sysroot/etc with Alpine sysroot/etc. To upstream, I will have to change the build scripts to supports both.

@mraleph PR #51044 is up. I explicitly split it into 4 commits hope for an easier review, but looks like Gerrit just combine them as one. So I will briefly explain here:

Add script for downloading alpine sysroot. Replace boolean flag dart_use_debian_sysroot with string flag dart_use_sysroot, which can be "", "alpine" or "debian". Setup compiler target & flags for dart_use_sysroot == "alpine".
Force use alpine's host clang when dart_use_sysroot == "alpine", as Google's provided clang doesn't work.
No -Werror when dart_use_sysroot == "alpine", as musl has different headers than glibc which caused quite a lot warnings.
Fix reinterpret_cast of NULL on musl by replace NULL with 0, tested on glibc as well and it seems fine.

Dart with tcmalloc can be built against on musl with a proper re-configure described in the documentation: https://github.com/dart-lang/sdk/blob/main/third_party/tcmalloc/README.dart
The document mentioned:
Make sure that include/config.h defines HAVE_UCONTEXT_H on Linux,

However, using the provided configure_command does not define HAVE_UCONTEXT_H: https://github.com/dart-lang/sdk/blob/main/third_party/tcmalloc/configure_command
It turns out --enable-cpu-profiler is required to run AC_PC_FROM_UCONTEXT, which will define HAVE_UCONTEXT_H:
https://github.com/gperftools/gperftools/blob/bf8b714bf5075d0a6f2f28504b43095e2b1e11c5/configure.ac#L270-L273


How to build a magic mirror

I bet a lot of people coming to your site for the first time are wondering "how do I build a magic mirror?" And all they will find is software documentation. I'd recommend just putting a link to your blog post or something near the top to send people towards the end product so they can understand the big picture.

I'd send them to the forums,
https://forum.magicmirror.builders/category/12/show-your-mirror
there are thousands of ways.
I don't use a mirrored surface.. don't NEED another 'mirror' in the house.. but the info is what I want

I'd recommend just putting a link to your blog post or something near the top

Do you mean the start site of this github repo or the documentation or ...?

Either one - just hit the link and imagine you've come to these pages for the first time, because you heard about this concept of magic mirrors and have no idea where to start 😀


Draft version?

Hi,
Just curious about which draft version is implemented by this library.
For instance the ruby one listed on http://jwt.io clearly states it's implementing draft 6.
Putting it in the README as they do might help.
Thanks!

Great question.  I'm currently working on an update that includes support for JWK and JWE.  I'll make sure to document draft numbers.

TODO: verify the 3.0.0 implementation complies with the latest version of the spec and add that information to README

resolved by<PHONE_NUMBER>e7c0b347923a9ffc13799c80b72ce4


Facing microk8s is not running issue.No resources found in default namespace showing for  microk8s.kubectl get  nodes

I am trying to set up microk8s multi-node cluster on my Centos machine. But when i run  microk8s.status it's showing microk8s is not running. Use microk8s inspect for a deeper inspection and For microk8s.kubectl get  nodes is showing  No resources found in default namespace. The output of microk8s inspect is
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

 WARNING:  IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT
The change can be made persistent with: sudo apt-get install iptables-persistent
Building the report tarball

snap    <IP_ADDRESS>-1.el7
snapd   <IP_ADDRESS>-1.el7
series  16
centos  7
kernel  3.10.0-1062.12.1.el7.x86_64

I am using
microk8s  v1.18.8      1609  1.18/stable    canonical✓  classic
Attaching microk8s.inspect tarball
inspection-report-20200915_045010.tar.gz

@ppratheesh thanks for using MicroK8s.  Looking at the logs, it seems like you had previously installed a 1.19 version and then finally decided to go for the 1.18/stable.
Im interested to know how you reverted to 1.18?
Did you do a sudo snap remove microk8s? If no, can you try to sudo snap remove microk8s followed by sudo snap install microk8s --channel 1.18/stable --classic

Hi @balchua ,I did the same way , i removed the microk8s using   snap remove microk8s and  installed it using microk8s --channel 1.18/stable --classic

Hi @balchua , I did the same way , installed it using snap remove microk8s and installed it using snap install microk8s --channel 1.18/stable --classic

@ppratheesh thanks.
Couple of things.  The warning that shows up when you run microk8s inspect
WARNING:  IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT
The change can be made persistent with: sudo apt-get install iptables-persistent

Can you try that?
Second, im taking a wild guess here, it seems like the apiserver is listening to ipv6.  Is there a way to turn it off on your end? Just to see if it is causing it to get undetected.  Once you manage to turn off ipv6, stop and start microk8s.
The weird thing is, the node appz isn't getting added to etcd.

Hi @balchua  I have tried with iptables -P FORWARD ACCEPT . No luck for me . we have 2 interfaces eh0 and eth1 . eth0 is the default interface .But eth1 accept external traffic
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         <IP_ADDRESS>        <IP_ADDRESS>         UG    100    0        0 eth0
<IP_ADDRESS>        <IP_ADDRESS>         <IP_ADDRESS>   U     100    0        0 eth0
<IP_ADDRESS>       <IP_ADDRESS>         <IP_ADDRESS>   U     101    0        0 eth1
<IP_ADDRESS>      <IP_ADDRESS>         <IP_ADDRESS>     U     0      0        0 docker0

This has something to do with etcd and api-server ?

@balchua  I think microk8s is using eth0, but node communication is possible only through eth1 . Any way to force microk8s to use eth1 instead of eth0

Hi @ppratheesh, looking at the containerd logs (journalctl -u snap.microk8s.daemon-containerd) there is this error:
Sep 15 03:39:32 appz1 microk8s.daemon-containerd[17681]: time="2020-09-15T03:39:32.531771263-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"

It seems your file system is XFS, right? This may play a role but I am not sure. In the kubelet logs (journalctl -u snap.microk8s.daemon-kubelet) there is this:
Sep 15 04:50:03 appz1 microk8s.daemon-kubelet[3964]: E0915 04:50:03.717268    3964 remote_runtime.go:81] Version from runtime service failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
Sep 15 04:50:03 appz1 microk8s.daemon-kubelet[3964]: E0915 04:50:03.717333    3964 kuberuntime_manager.go:197] Get runtime version failed: get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
Sep 15 04:50:03 appz1 microk8s.daemon-kubelet[3964]: F0915 04:50:03.717374    3964 server.go:274] failed to run Kubelet: failed to create kubelet: get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

To change the interface the CNI will use you can look at https://github.com/ubuntu/microk8s/issues/1554#issuecomment-691426908 but at this point I do not think this will make a difference because kubelet does not register the node at all.

Hi @ktsakalozos  i have docker installed separately on all machines .Do we face any conflicts because of that ?

Do we face any conflicts because of that ?

It is possible. Could you do a quick test. Stop dockerd and containerd services and restart MicroK8s with microk8s stop; microk8s start ? We may be able to rule out this kind of interference.

Hi @ktsakalozos
here is the log of containerd..
-- Logs begin at Mon 2020-09-14 21:45:19 PDT, end at Wed 2020-09-16 03:41:22 PDT. --
Sep 14 23:55:40 appz3 systemd[1]: Stopping Service for snap application microk8s.daemon-containerd...
Sep 14 23:55:40 appz3 systemd[1]: Stopped Service for snap application microk8s.daemon-containerd.
Sep 15 00:12:01 appz3 systemd[1]: Started Service for snap application microk8s.daemon-containerd.
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + export PATH=/snap/microk8s/1609/usr/sbin:/snap/microk8s/1609/usr/bin:/snap/microk8s/1609/sbin:/snap/microk8s/1609/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + PATH=/snap/microk8s/1609/usr/sbin:/snap/microk8s/1609/usr/bin:/snap/microk8s/1609/sbin:/snap/microk8s/1609/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ /snap/microk8s/1609/bin/uname -m
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + ARCH=x86_64
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + export LD_LIBRARY_PATH=:/snap/microk8s/1609/lib:/snap/microk8s/1609/usr/lib:/snap/microk8s/1609/lib/x86_64-linux-gnu:/snap/microk8s/1609/usr/lib/x86_64-linux-gnu
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + LD_LIBRARY_PATH=:/snap/microk8s/1609/lib:/snap/microk8s/1609/usr/lib:/snap/microk8s/1609/lib/x86_64-linux-gnu:/snap/microk8s/1609/usr/lib/x86_64-linux-gnu
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/1609/lib:/snap/microk8s/1609/usr/lib:/snap/microk8s/1609/lib/x86_64-linux
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/1609/lib:/snap/microk8s/1609/usr/lib:/snap/microk8s/1609/lib/x86_64-linux-gnu:/s
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + export XDG_RUNTIME_DIR=/var/snap/microk8s/common/run
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + XDG_RUNTIME_DIR=/var/snap/microk8s/common/run
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + mkdir -p /var/snap/microk8s/common/run
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + source /snap/microk8s/1609/actions/common/utils.sh
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + '[' -d /etc/apparmor.d ']'
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + app=containerd
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + '[' -e /var/snap/microk8s/1609/var/lock/gpu ']'
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + RUNTIME=runc
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ stat -f -c %T /var/snap/microk8s/common
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + FSTYPE=xfs
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + '[' xfs = zfs ']'
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + SNAPSHOTTER=overlayfs
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + sed 's@${SNAP}@/snap/microk8s/1609@g;s@${SNAP_DATA}@/var/snap/microk8s/1609@g;s@${SNAPSHOTTER}@overlayfs@g;s@${RUNTIME}@runc@g' /var/snap/microk8s/1609/args/containerd-template.t
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ is_service_expected_to_start flanneld
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ local service=flanneld
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ '[' -f /var/snap/microk8s/1609/var/lock/no-flanneld ']'
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ echo 1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + run_flanneld=1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + '[' 1 == 1 ']'
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + sed 's@${SNAP}@/snap/microk8s/1609@g;s@${SNAP_DATA}@/var/snap/microk8s/1609@g;s@${SNAP_COMMON}@/var/snap/microk8s/common@g' /var/snap/microk8s/1609/args/flannel-template.conflist
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ cat /var/snap/microk8s/1609/args/containerd
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + declare -a 'args=(--config ${SNAP_DATA}/args/containerd.toml
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: --root ${SNAP_COMMON}/var/lib/containerd
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: --state ${SNAP_COMMON}/run/containerd
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: --address ${SNAP_COMMON}/run/containerd.sock)'
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + set -a
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + . /var/snap/microk8s/1609/args/containerd-env
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ ulimit -n 65536
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: ++ ulimit -l 16384
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + set +a
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + n=0
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + '[' 0 -ge 20 ']'
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + grep default
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + ip route
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + break
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + export CILIUM_SOCK=/var/snap/microk8s/1609/var/run/cilium/cilium.sock
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + CILIUM_SOCK=/var/snap/microk8s/1609/var/run/cilium/cilium.sock
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: + exec /snap/microk8s/1609/bin/containerd --config /var/snap/microk8s/1609/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/commo
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.737544968-07:00" level=info msg="starting containerd" revision=bb71b10fd8f58240ca47fbb579b9d1028eea7c84 version=v1.2.5
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.738096552-07:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.738137734-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.738499871-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/snap/microk8s/common/var/lib/containerd/io.
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.738524315-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.748924603-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs no
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.748965180-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.749021107-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.749273112-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="/var/snap/microk8s/common/var/lib/containerd/io.c
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.749295202-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.749567776-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/snap/microk8s/common/var/lib/containerd/io.co
Sep 15 00:12:01 appz3 microk8s.daemon-containerd[16878]: time="2020-09-15T00:12:01.749587483-07:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1

cant figure out...why the nodes are not joining to master..please lemme know if any

@ktsakalozos
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

 WARNING:  IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT
The change can be made persistent with: sudo apt-get install iptables-persistent
Building the report tarball
  Report tarball is at /var/snap/microk8s/1609/inspection-report-20200916_034439.tar.gz
[appz@appz3 ~]$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.

[appz@appz1 args]$ sudo systemctl stop dockerd
Failed to stop dockerd.service: Unit dockerd.service not loaded.
[appz@appz1 args]$ sudo systemctl stop docker
[appz@appz1 args]$ sudo systemctl status containerd
Unit containerd.service could not be found.
[appz@appz1 args]$ sudo systemctl status container
Unit container.service could not be found.
[appz@appz1 args]$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2020-09-16 03:33:06 PDT; 3min 18s ago
     Docs: https://docs.docker.com
  Process: 28119 ExecStart=/usr/bin/dockerd (code=exited, status=0/SUCCESS)
 Main PID: 28119 (code=exited, status=0/SUCCESS)
[appz@appz1 args]$ microk8s stop
Stopped.
[appz@appz1 args]$ microk8s start
Started.
Enabling pod scheduling
[appz@appz1 args]$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.

Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/1609/inspection-report-20200916_033850.tar.gz

inspection-report-20200916_033850.tar.gz
Is there any way to start the microk8s using a specific interface . Now I'm guessing it uses eth0 .

-- Logs begin at Mon 2020-09-14 23:21:51 PDT, end at Wed 2020-09-16 04:00:51 PDT. --
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984495891-07:00" level=info msg="containerd successfully booted in 0.018532s"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984475305-07:00" level=info msg=serving... address="/var/snap/microk8s/common/run/con
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984330011-07:00" level=info msg=serving... address="<IP_ADDRESS>:1338"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983809352-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983789440-07:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983608688-07:00" level=info msg="Connect containerd service"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983572127-07:00" level=info msg="Start cri plugin with config {PluginConfig:{Containe
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983370152-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983351935-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." t
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983331634-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." typ
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983307980-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..."
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983246720-07:00" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." t
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983224605-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"...
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983203742-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." ty
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983184921-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." ty
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983165685-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983147282-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." ty
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983127122-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983107261-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983084226-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982993340-07:00" level=info msg="loading plugin "io.containerd.internal.v1.restart"..
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982940522-07:00" level=info msg="loading plugin "io.containerd.service.v1.tasks-servi
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982365715-07:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982275590-07:00" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982169962-07:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982144338-07:00" level=info msg="loading plugin "io.containerd.service.v1.snapshots-s
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982124028-07:00" level=info msg="loading plugin "io.containerd.service.v1.namespaces-
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982103336-07:00" level=info msg="loading plugin "io.containerd.service.v1.leases-serv
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982079530-07:00" level=info msg="loading plugin "io.containerd.service.v1.images-serv
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982036294-07:00" level=info msg="loading plugin "io.containerd.service.v1.diff-servic
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982011638-07:00" level=info msg="loading plugin "io.containerd.service.v1.content-ser
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981979552-07:00" level=info msg="loading plugin "io.containerd.service.v1.containers-
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981834060-07:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981807227-07:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981607746-07:00" level=warning msg="could not use snapshotter btrfs in metadata plugi
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981595972-07:00" level=warning msg="could not use snapshotter zfs in metadata plugin"
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981583734-07:00" level=warning msg="could not use snapshotter overlayfs in metadata p
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981567887-07:00" level=warning msg="could not use snapshotter aufs in metadata plugin
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981539486-07:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981518875-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981239868-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"...
...skipping...
-- Logs begin at Mon 2020-09-14 23:21:51 PDT, end at Wed 2020-09-16 04:00:51 PDT. --
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984495891-07:00" level=info msg="containerd successfully booted in 0.018532s"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984475305-07:00" level=info msg=serving... address="/var/snap/microk8s/common/run/con
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984330011-07:00" level=info msg=serving... address="<IP_ADDRESS>:1338"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983809352-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983789440-07:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983608688-07:00" level=info msg="Connect containerd service"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983572127-07:00" level=info msg="Start cri plugin with config {PluginConfig:{Containe
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983370152-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983351935-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." t
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983331634-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." typ
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983307980-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..."
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983246720-07:00" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." t
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983224605-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"...
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983203742-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." ty
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983184921-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." ty
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983165685-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983147282-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." ty
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983127122-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983107261-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983084226-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982993340-07:00" level=info msg="loading plugin "io.containerd.internal.v1.restart"..
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982940522-07:00" level=info msg="loading plugin "io.containerd.service.v1.tasks-servi
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982365715-07:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982275590-07:00" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982169962-07:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982144338-07:00" level=info msg="loading plugin "io.containerd.service.v1.snapshots-s
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982124028-07:00" level=info msg="loading plugin "io.containerd.service.v1.namespaces-
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982103336-07:00" level=info msg="loading plugin "io.containerd.service.v1.leases-serv
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982079530-07:00" level=info msg="loading plugin "io.containerd.service.v1.images-serv
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982036294-07:00" level=info msg="loading plugin "io.containerd.service.v1.diff-servic
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982011638-07:00" level=info msg="loading plugin "io.containerd.service.v1.content-ser
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981979552-07:00" level=info msg="loading plugin "io.containerd.service.v1.containers-
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981834060-07:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981807227-07:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981607746-07:00" level=warning msg="could not use snapshotter btrfs in metadata plugi
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981595972-07:00" level=warning msg="could not use snapshotter zfs in metadata plugin"
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981583734-07:00" level=warning msg="could not use snapshotter overlayfs in metadata p
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981567887-07:00" level=warning msg="could not use snapshotter aufs in metadata plugin
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981539486-07:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981518875-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981239868-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"...
[appz@appz1 args]$
[appz@appz1 args]$ sudo journalctl -ru snap.microk8s.daemon-containerd
-- Logs begin at Mon 2020-09-14 23:21:51 PDT, end at Wed 2020-09-16 04:01:07 PDT. --
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984495891-07:00" level=info msg="containerd successfully booted in 0.018532s"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984475305-07:00" level=info msg=serving... address="/var/snap/microk8s/common/run/con
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.984330011-07:00" level=info msg=serving... address="<IP_ADDRESS>:1338"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983809352-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983789440-07:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983608688-07:00" level=info msg="Connect containerd service"
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983572127-07:00" level=info msg="Start cri plugin with config {PluginConfig:{Containe
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983370152-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983351935-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." t
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983331634-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." typ
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983307980-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..."
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983246720-07:00" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." t
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983224605-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"...
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983203742-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." ty
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983184921-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." ty
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983165685-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..
Sep 16 03:36:46 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983147282-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." ty
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983127122-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983107261-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.983084226-07:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982993340-07:00" level=info msg="loading plugin "io.containerd.internal.v1.restart"..
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982940522-07:00" level=info msg="loading plugin "io.containerd.service.v1.tasks-servi
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982365715-07:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982275590-07:00" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982169962-07:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982144338-07:00" level=info msg="loading plugin "io.containerd.service.v1.snapshots-s
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982124028-07:00" level=info msg="loading plugin "io.containerd.service.v1.namespaces-
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982103336-07:00" level=info msg="loading plugin "io.containerd.service.v1.leases-serv
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982079530-07:00" level=info msg="loading plugin "io.containerd.service.v1.images-serv
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982036294-07:00" level=info msg="loading plugin "io.containerd.service.v1.diff-servic
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.982011638-07:00" level=info msg="loading plugin "io.containerd.service.v1.content-ser
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981979552-07:00" level=info msg="loading plugin "io.containerd.service.v1.containers-
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981834060-07:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." t
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981807227-07:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981607746-07:00" level=warning msg="could not use snapshotter btrfs in metadata plugi
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981595972-07:00" level=warning msg="could not use snapshotter zfs in metadata plugin"
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981583734-07:00" level=warning msg="could not use snapshotter overlayfs in metadata p
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981567887-07:00" level=warning msg="could not use snapshotter aufs in metadata plugin
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981539486-07:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..."
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981518875-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981239868-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.981216241-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.980984715-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlay
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.980910297-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.980872094-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.975631134-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.975599836-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.975345969-07:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs".
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.975294861-07:00" level=info msg="loading plugin "io.containerd.content.v1.content"...
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: time="2020-09-16T03:36:45.974668798-07:00" level=info msg="starting containerd" revision=bb71b10fd8f58240ca47fb
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + exec /snap/microk8s/1609/bin/containerd --config /var/snap/microk8s/1609/args/containerd.toml --root /var/sna
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + CILIUM_SOCK=/var/snap/microk8s/1609/var/run/cilium/cilium.sock
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + export CILIUM_SOCK=/var/snap/microk8s/1609/var/run/cilium/cilium.sock
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + break
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + grep default
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + ip route
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + '[' 0 -ge 20 ']'
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + n=0
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + set +a
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ ulimit -l 16384
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ ulimit -n 65536
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + . /var/snap/microk8s/1609/args/containerd-env
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + set -a
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: --address ${SNAP_COMMON}/run/containerd.sock)'
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: --state ${SNAP_COMMON}/run/containerd
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: --root ${SNAP_COMMON}/var/lib/containerd
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + declare -a 'args=(--config ${SNAP_DATA}/args/containerd.toml
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ cat /var/snap/microk8s/1609/args/containerd
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + sed 's@${SNAP}@/snap/microk8s/1609@g;s@${SNAP_DATA}@/var/snap/microk8s/1609@g;s@${SNAP_COMMON}@/var/snap/micr
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + '[' 1 == 1 ']'
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + run_flanneld=1
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ echo 1
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ '[' -f /var/snap/microk8s/1609/var/lock/no-flanneld ']'
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ local service=flanneld
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ is_service_expected_to_start flanneld
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + sed 's@${SNAP}@/snap/microk8s/1609@g;s@${SNAP_DATA}@/var/snap/microk8s/1609@g;s@${SNAPSHOTTER}@overlayfs@g;s@
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + SNAPSHOTTER=overlayfs
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + '[' xfs = zfs ']'
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + FSTYPE=xfs
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ stat -f -c %T /var/snap/microk8s/common
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + RUNTIME=runc
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + '[' -e /var/snap/microk8s/1609/var/lock/gpu ']'
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + app=containerd
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + '[' -d /etc/apparmor.d ']'
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + source /snap/microk8s/1609/actions/common/utils.sh
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + mkdir -p /var/snap/microk8s/common/run
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + XDG_RUNTIME_DIR=/var/snap/microk8s/common/run
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + export XDG_RUNTIME_DIR=/var/snap/microk8s/common/run
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/1609/lib:/s
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/1609
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + LD_LIBRARY_PATH=:/snap/microk8s/1609/lib:/snap/microk8s/1609/usr/lib:/snap/microk8s/1609/lib/x86_64-linux-gnu
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + export LD_LIBRARY_PATH=:/snap/microk8s/1609/lib:/snap/microk8s/1609/usr/lib:/snap/microk8s/1609/lib/x86_64-li
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + ARCH=x86_64
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: ++ /snap/microk8s/1609/bin/uname -m
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + PATH=/snap/microk8s/1609/usr/sbin:/snap/microk8s/1609/usr/bin:/snap/microk8s/1609/sbin:/snap/microk8s/1609/bi
Sep 16 03:36:45 appz1 microk8s.daemon-containerd[29803]: + export PATH=/snap/microk8s/1609/usr/sbin:/snap/microk8s/1609/usr/bin:/snap/microk8s/1609/sbin:/snap/microk8s/
Sep 16 03:36:45 appz1 systemd[1]: Started Service for snap application microk8s.daemon-containerd.

Hi @ktsakalozos
[appz@appz1 cni-network]$ microk8s kubectl get pods -A
No resources found
[appz@appz1 cni-network]$ microk8s kubectl get all
NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   <IP_ADDRESS>   <none>        443/TCP   5h18m
[appz@appz1 cni-network]$ microk8s kubectl get node
No resources found in default namespace.

Ah, I see now why the focus on the network! The <IP_ADDRESS>/16 is the pods network but it is also the IP of eth0. @balchua spotted this way before me!
@ppratheesh could you please offer another datapoint? Can you snap install microk8s from the 1.19 channel and attaching a microk8s.inspect tarball? In the 1.19 channel we have a newer containerd and a different CNI (unfortunately also using the <IP_ADDRESS>/16 range).

Hi @ktsakalozos  please have a look at the outputs
``
[appz@appz1 ~]$ snap list
Name      Version    Rev   Tracking       Publisher   Notes
core      16-2.46.1  9993  latest/stable  canonical✓  core
microk8s  v1.19.0    1667  1.19/stable    canonical✓  classic
[appz@appz1 ~]$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
[appz@appz1 ~]$ microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Building the report tarball
Report tarball is at /var/snap/microk8s/1667/inspection-report-20200916_045123.tar.gz

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
inet <IP_ADDRESS>  netmask <IP_ADDRESS>  broadcast <IP_ADDRESS>
inet6 fe80::42:88ff:fe44:2bed  prefixlen 64  scopeid 0x20
ether 02:42:88:44:2b:ed  txqueuelen 0  (Ethernet)
RX packets 2214851  bytes 279749619 (266.7 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 2272652  bytes 908365619 (866.2 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet <IP_ADDRESS>  netmask <IP_ADDRESS>  broadcast <IP_ADDRESS>
inet6 fe80::52ca:79ca:b4e9:3433  prefixlen 64  scopeid 0x20
ether b2:61:6e:73:6c:14  txqueuelen 1000  (Ethernet)
RX packets 2864260  bytes<PHONE_NUMBER>1 (15.0 GiB)
RX errors 0  dropped 16  overruns 0  frame 0
TX packets 2368751  bytes 223097429 (212.7 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet <IP_ADDRESS>  netmask <IP_ADDRESS>  broadcast <IP_ADDRESS>
inet6 fe80::5054:ff:fe4b:63fc  prefixlen 64  scopeid 0x20
ether 52:54:00:4b:63:fc  txqueuelen 1000  (Ethernet)
RX packets 66189342  bytes<PHONE_NUMBER>5 (10.4 GiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 64092998  bytes<PHONE_NUMBER>8 (58.7 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
inet <IP_ADDRESS>  netmask <IP_ADDRESS>  broadcast <IP_ADDRESS>
inet6 fe80::246e:7fff:feb3:16a9  prefixlen 64  scopeid 0x20
ether 26:6e:7f:b3:16:a9  txqueuelen 0  (Ethernet)
RX packets 0  bytes 0 (0.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 0  bytes 0 (0.0 B)
TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet <IP_ADDRESS>  netmask <IP_ADDRESS>
inet6 ::1  prefixlen 128  scopeid 0x10
loop  txqueuelen 1000  (Local Loopback)
RX packets 439269790  bytes<PHONE_NUMBER>28 (115.8 GiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 439269790  bytes<PHONE_NUMBER>28 (115.8 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[inspection-report-20200916_045123.tar.gz](https://github.com/ubuntu/microk8s/files/5231921/inspection-report-20200916_045123.tar.gz)

It looks like the same issue as in https://github.com/rancher/k3s/issues/495#issuecomment-496366229 . Lets fix the xfs d_type=true before we address any networking issues we have. You could try to edit /var/snap/microk8s/current/args/containerd and /var/snap/microk8s/current/args/kubelet and point containerd in a directory on a different file system. Or I would recommend you reformat the directory already in used by snaps (/var/snap).

I thought that containerd log was benign.  Haha

Thanks a lot @ktsakalozos  . That worked. I have  reformatted the directory /var/snap and cluster is up now .. by the way i did the same on one of my node and node is able to join the cluster now .
NAME    STATUS   ROLES    AGE   VERSION
appz2   Ready    <none>   76m   v1.19.0-34+09a4aa08bb9e93
appz1   Ready    <none>   97m   v1.19.0-34+09a4aa08bb9e93

Thanks @ppratheesh for confirming. 👍

Thanks a lot @ktsakalozos . That worked. I have reformatted the directory /var/snap and cluster is up now .. by the way i did the same on one of my node and node is able to join the cluster now .
NAME    STATUS   ROLES    AGE   VERSION
appz2   Ready    <none>   76m   v1.19.0-34+09a4aa08bb9e93
appz1   Ready    <none>   97m   v1.19.0-34+09a4aa08bb9e93

@ppratheesh What exactly did you do to "reformat" the /var/snap directory? I am having the same problem and want to see if this resolves my issues.


When there are lots of packages to inspect a socket is used for each package

I've been trying out nukeeper against a large set of project files, which in total have ~390 unique nugets to check.
Basically running:
nukeeper inspect -v d -c patch
I've noticed that the first updates work well (however, it looks like a fresh SSL socket is consumed for every package being checked, which is quite a lot in this case)
However, second checks seem to be very much slower, especially when the data is in cache (in fact the cache is slower than checking the web server, suggesting that the cache code in the nuget api is serialized, breaking into the process suggests that the xml parsing might be locking)
So I figure that the problem is something underlying in the nuget api usage that stops socket reuse (eg is there perhaps an object in nuget that should be Idisposable (but isn't))
I noted from debugging that PackageMetadataResource from seems to have a http source hanging off it, which is generated from the owning source repository, however, there's one sourcerepository per call.
I thought I'd try and share the source repository between all users of the same packagesource.  This appears to have fixed the socket consumption, and caching problems.
It appears that making the code share one source repository per package source fixes the caching performance.  But I'm not sure if it's a "good" fix for all code paths, as it still shows the large socket usage in the uncached case of hitting the server for data.  I suspect that perhaps the internals of the nuget library aren't making the massive concurrency good
To improve the cached case, within PackageVersionsLookup.cs, add a:
using System.Collections.Concurrent;

And update BuildSourceRepository as:
private ConcurrentDictionary<PackageSource, SourceRepository> _sourceCache = new ConcurrentDictionary<PackageSource, SourceRepository>();
private SourceRepository BuildSourceRepository(PackageSource packageSource)
{
    return _sourceCache.GetOrAdd(packageSource,
        key => new SourceRepository(key, Repository.Provider.GetCoreV3()));
}

The change pass the unit tests, but I'm not sure if it's safe for general usage or not.
I'm actually surprised that the code isn't hitting a http request limit within the .net client code, so it could be I've changed/broken something in my .net config files (but I don't recall changing anything for .net core)

Good spot! This will need some investigation - possibly how NuGet itself is using this code could point to some mistakes we've made.
On the request limit - I'm reasonably sure dotnetcore removed those. HttpClientHandler.MaxConnectionsPerServer is set to Int32.MaxValue by default.

Ah, that answers why I hit the internal nuget server with so many connections :)  I've not had to use .net core that much, yet. I know that we had to tune up the numbers for .net framework from the default of 4, but normally just in api callers.
Overall this looks like a great tool, having written a posh script to try and do this, that would take 20 minutes on the 300+ projects (as it ran nuget.exe on every proj file), this takes far less time to inspect.  And compared to Visual studio, that was 2-3 hours to do the nuget updates....

I am going to try a variation on this fix. I think it's better to make the PackageSource -> SourceRepository map upfront. In many cases it will contain only 1 element, and even with internal sources, there will only be 2 or 3 of them most times.

@cg110 Are you set up to run the source code of a PR? I hope that #384 fixes the cache performance by reusing SourceRepository instances across parallel calls to  PackageVersionsLookup - it's basically your code and is IMHO safe.
I assume that "~390 unique nugets" means 390 packages from a handful of internal and public package sources?

Change has been merged

Ok, I'll grab the latest master, and give it a go.  For reference the inspect times with the 0.6.1 version running:
nukeeper inspect -v d -c patch -s https://internalnugetserver/nuget/

First (hits web server) inspect: 10s
Second (cached) inspect: 2m18s
In terms of solution/project sizes:
Found 12728 packages in use, 321 distinct, in 340 projects.
I'm not sure where I got 390 from, as it's not quite that many!  And they're from internal, and external sources.

Do you want a separate bug on the socket consumption (I realise my original description was unclear, in the case the cache is hit there is no socket used to talk to the web server)
In the first inspect there appears to be a https connection per package (which is probably not a friendly thing to do to the web servers)
The change for this has fixed the second inspect where there's cache locking problems, and the performance was appalling (I don't know why, but I assume it's something to do with parsing cache files)
I was wondering if PackageMetadataResource needs to be cached, but I can't tell if it's thread safe or not to do so.

Hmm, looks like the core nuget update has code to restrict it to running maxTasks (4) tasks at a time:
https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.PackageManagement/NuGetPackageManager.cs#L670
Perhaps something similar in BulkPackageLookup.FindVersionUpdates would also work.  I'll see if I can throw something together.

Version using a task limit (sorry I don't have github setup locally to make a proper pull request):
All within BulkPackageLookup.cs, add a new const:
// Possibly tune this further, 10 stops the code hitting servers with 1 connection per package
// nuget internally looks to use 4, but this seemed quite a bit slower.
private const int maxNugetApiTasks = 10;

Then update the code to:
public async Task<IDictionary<string, PackageLookupResult>> FindVersionUpdates(
    IEnumerable<PackageIdentity> packages,
    NuGetSources sources,
    VersionChange allowedChange)
{
    var tasks = new List<Task<PackageLookupResult>>(maxNugetApiTasks);

    var latestOfEach = packages
        .GroupBy(pi => pi.Id.ToLowerInvariant())
        .Select(HighestVersion);

    var result = new Dictionary<string, PackageLookupResult>(StringComparer.OrdinalIgnoreCase);

    foreach (var latestPackage in latestOfEach)
    {
        if (tasks.Count >= maxNugetApiTasks)
        {
            ProcessLookupResult(await CompleteAndRemoveAnyTask(tasks), result);
        }

        var lookupTask = _packageLookup.FindVersionUpdate(latestPackage, sources, allowedChange);
        tasks.Add(lookupTask);
    }

    foreach (var lookupResult in await Task.WhenAll(tasks))
    {
        ProcessLookupResult(lookupResult, result);
    }

    return result;
}

private async Task<T> CompleteAndRemoveAnyTask<T>(List<Task<T>> tasks)
{
    var completeTask = await Task.WhenAny(tasks);
    tasks.Remove(completeTask);
    return await completeTask;
}

The code is now limited to 10 version check tasks, and so 10 nuget connections/ssl sockets per source.  Note that this does potentially mean that we'll run at the speed of the slowest nuget source, but I suspect that was always the case, and in fact this may mean far less tasks for .net to manage, and possibly even some memory savings to be had.

the speed of the slowest nuget source

That can be bad, to be honest. I often have internal source set up, then I take the laptop home and they're just not reachable, and time out.

I suspect right now with all 300 requests firing off at the same time, you probably get them all timeout about the same time.  This change could make that 300/10 * the timeout length :(  I guess more smarts would be needed to detect a source was not actually available.

Released today

Just to confirm the performance change with 0.6.2.  The first run is took ~8.6s, the second (cached) run is ~5.8s.
So much improved, still a spike in connections for the first uncached case, but that's expected.


Only the last referenced document of an array is clickable

Describe the bug
I have a mongoose model Thing with a property users. This is an array of objects with each a ref to User using an ObjectId.
In the front-end this translates nicely to a link with the User's email address, and I can click it to view that user.
However only the last User of the array is resolved, the others show a non-clickable ObjectId.
Installed libraries and their versions

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Expected behavior
All refs should be resolved.

Can you check if the problem still exists in the latest version of admin-bro & @admin-bro/mongoose & @admin-bro/expressjs? 2.0.1 is a bit outdated

Sorry, I mentioned the incorrect versions.
    "@admin-bro/express": "^3.0.1",
    "@admin-bro/mongoose": "^1.1.0",
    "admin-bro": "^3.3.1",

Could you share your mongoose schemas? I'll try to reproduce/fix it sometime this week

Sure!
module.exports = mongoose.model('User', {
  email: String,
  firstname: String,
  lastname: String,
  roles: [String],
})

module.exports = mongoose.model('Thing', {
  name: String,
  users: [{
    role: String,
    userId: {
      type: mongoose.ObjectId,
      ref: 'User',
    },
  }],
})

@WeeJeWel The fix is included in<EMAIL_ADDRESS>so I'm closing this issue. Sorry for the delay in fixing this.

Thank you!

I'm sorry, but it doesn't seem to be solved for me using<EMAIL_ADDRESS>

I'll check this again next week but I reproduced the problem using your schema and this seemed to have fixed it. Have you added any custom property options in resource options?


error: winss/premake5.lua:28: attempt to call a nil value (field 'getn')

I Just followed your README.md instruction to make vs2015 project.
But it just spit out error "Error: ~~~~~~/winss/premake5.lua:28: attempt to call a nil value (field 'getn')"
I don't know what was wrong.

The function call is deprecated and is probably removed in the later versions. The build is using https://github.com/premake/premake-core/releases/download/v5.0.0-alpha10/premake-5.0.0-alpha10-windows.zip if you want to try that?
I will update the premake script to be compatible with lastest version.


linting  in activation functions

linting issues solved in activation layers.10/10 for all activation functions #77

@jeffin07 what is the score that you are getting on layers from pylint?

@imdeepmind  10/10


Support for 2018 Chevrolet Equinox Premier With Driver Confidence and Convenience Package

Support for 2018 Chevrolet Equinox Premier With Driver Confidence and Convenience Package?

Support for this car will be added through community development. Feel free to join the GM channel on our community discord, discord.comma.ai, to get started