Internal Load Balancer not automatically adding target network IP configurations

Is this a request for help?:
Yes
Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE
What version of acs-engine?:
0.12.1

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.9.1
What happened:
Internal Load Balancer created and associated with availability set but no target network ip configurations
were added. Manually adding them in the portal is a functional workaround.
What you expected to happen:
Minimally, the target ip configuration of the load balancer endpoints should be added.
How to reproduce it (as minimally and precisely as possible):

Deploy acs kubernetes cluster with single master, single agentpool.
Use "availabilityProfile": "AvailabilitySet"

Create Kubernetes service of type LoadBalancer, annotate with "service.beta.kubernetes.io/azure-load-balancer-internal=true"

Observe no target network ip configurations were added to the backend pool. Test to confirm that traffic does not flow as expected.

Anything else we need to know:
Deployed to custom vnet

@feiskyer Do you have any ideas if your fix is coming in v1.9.3? Thanks for that btw!

@jackfrancis Cherry pick to v1.9 branch has been merged. It will included in v1.9.3

With kubernetes v1.9.3 also, I am still facing this problem.
At the time of writing, the latest acs-engine release version: v0.12.5 does not supporting creating kubernetes 1.9.3 cluster. I built the acs-engine from the master branch commit fe571504.
./acs-engine version

Version: canary
GitCommit: fe57150
GitTreeState: clean

When a kubernetes service was created with this service.beta.kubernetes.io/azure-load-balancer-internal: "true" annotation, a load balancer with private ip was created but no target network ip configurations were added to the backend pool.

@diwakar-s-maurya Could you check logs of kube-controller-manger and check whether there is something wrong happened? e.g. by
PODNAME=$(kubectl -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system logs $PODNAME --tail 500

@feiskyer These are the log. Creating internal load balancer type service does not automatically add target network IP configurations.
Logs after almost 5 mins of fresh cluster v1.9.3 creation by acs-engine fe57150
I0211 05:50:10.395429       1 feature_gate.go:190] feature gates: map[ServiceNodeExclusion:true]
I0211 05:50:10.395516       1 controllermanager.go:108] Version: v1.9.3
I0211 05:50:10.403199       1 leaderelection.go:174] attempting to acquire leader lease...
E0211 05:50:10.403609       1 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp <IP_ADDRESS>:443: getsockopt: connection refused
I0211 05:50:23.324910       1 leaderelection.go:184] successfully acquired lease kube-system/kube-controller-manager
I0211 05:50:23.333035       1 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"7387e061-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"17", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s-master-40117112-0 became leader
E0211 05:50:23.347425       1 controllermanager.go:386] Server isn't healthy yet.  Waiting a little while.
E0211 05:50:24.361405       1 controllermanager.go:386] Server isn't healthy yet.  Waiting a little while.
I0211 05:50:25.547239       1 azure.go:249] azure: using client_id+client_secret to retrieve access token
I0211 05:50:27.488072       1 controllermanager.go:483] Starting "replicationcontroller"
I0211 05:50:27.488131       1 controller_utils.go:1019] Waiting for caches to sync for tokens controller
I0211 05:50:27.588702       1 controller_utils.go:1026] Caches are synced for tokens controller
I0211 05:50:27.785005       1 controllermanager.go:493] Started "replicationcontroller"
I0211 05:50:27.785032       1 controllermanager.go:483] Starting "service"
I0211 05:50:27.785639       1 replica_set.go:183] Starting replicationcontroller controller
I0211 05:50:27.785782       1 controller_utils.go:1019] Waiting for caches to sync for ReplicationController controller
I0211 05:50:27.830834       1 controllermanager.go:493] Started "service"
I0211 05:50:27.830856       1 controllermanager.go:483] Starting "ttl"
I0211 05:50:27.830967       1 service_controller.go:187] Starting service controller
I0211 05:50:27.830983       1 controller_utils.go:1019] Waiting for caches to sync for service controller
I0211 05:50:27.872271       1 controllermanager.go:493] Started "ttl"
I0211 05:50:27.872339       1 controllermanager.go:483] Starting "route"
I0211 05:50:27.872362       1 core.go:132] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0211 05:50:27.872441       1 controllermanager.go:490] Skipping "route"
I0211 05:50:27.872477       1 controllermanager.go:483] Starting "pvc-protection"
W0211 05:50:27.872492       1 controllermanager.go:490] Skipping "pvc-protection"
I0211 05:50:27.872516       1 controllermanager.go:483] Starting "resourcequota"
I0211 05:50:27.872414       1 ttl_controller.go:116] Starting TTL controller
I0211 05:50:27.872644       1 controller_utils.go:1019] Waiting for caches to sync for TTL controller
I0211 05:50:27.983935       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions ingresses}
I0211 05:50:27.983986       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions replicasets}
I0211 05:50:27.984032       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { endpoints}
I0211 05:50:27.984131       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps controllerrevisions}
I0211 05:50:27.984174       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps daemonsets}
I0211 05:50:27.984240       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {networking.k8s.io networkpolicies}
W0211 05:50:27.984261       1 shared_informer.go:310] resyncPeriod<PHONE_NUMBER>6268 is smaller than resyncCheckPeriod<PHONE_NUMBER>9038 and the informer has already started. Changing it to<PHONE_NUMBER>9038
I0211 05:50:27.984341       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { serviceaccounts}
I0211 05:50:27.984449       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { limitranges}
I0211 05:50:27.984529       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps statefulsets}
I0211 05:50:27.984602       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {events.k8s.io events}
I0211 05:50:27.984641       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {batch cronjobs}
I0211 05:50:27.984785       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions daemonsets}
I0211 05:50:27.984840       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { podtemplates}
I0211 05:50:27.984945       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps deployments}
I0211 05:50:27.985018       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {batch jobs}
I0211 05:50:27.985052       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {rbac.authorization.k8s.io roles}
I0211 05:50:27.985122       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {rbac.authorization.k8s.io rolebindings}
I0211 05:50:27.985152       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps replicasets}
I0211 05:50:27.985232       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {autoscaling horizontalpodautoscalers}
I0211 05:50:27.985268       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {policy poddisruptionbudgets}
W0211 05:50:27.985324       1 shared_informer.go:310] resyncPeriod<PHONE_NUMBER>6377 is smaller than resyncCheckPeriod<PHONE_NUMBER>9038 and the informer has already started. Changing it to<PHONE_NUMBER>9038
I0211 05:50:27.985438       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions deployments}
I0211 05:50:27.985531       1 controllermanager.go:493] Started "resourcequota"
I0211 05:50:27.985551       1 controllermanager.go:483] Starting "daemonset"
I0211 05:50:27.985846       1 resource_quota_controller.go:272] Starting resource quota controller
I0211 05:50:27.985862       1 controller_utils.go:1019] Waiting for caches to sync for resource quota controller
I0211 05:50:27.986172       1 resource_quota_monitor.go:289] QuotaMonitor running
I0211 05:50:28.092787       1 controllermanager.go:493] Started "daemonset"
I0211 05:50:28.092853       1 controllermanager.go:483] Starting "job"
I0211 05:50:28.092997       1 daemon_controller.go:232] Starting daemon sets controller
I0211 05:50:28.093029       1 controller_utils.go:1019] Waiting for caches to sync for daemon sets controller
I0211 05:50:28.135334       1 controllermanager.go:493] Started "job"
I0211 05:50:28.135475       1 controllermanager.go:483] Starting "csrcleaner"
I0211 05:50:28.135657       1 job_controller.go:138] Starting job controller
I0211 05:50:28.135693       1 controller_utils.go:1019] Waiting for caches to sync for job controller
I0211 05:50:28.180672       1 controllermanager.go:493] Started "csrcleaner"
I0211 05:50:28.180726       1 controllermanager.go:483] Starting "attachdetach"
I0211 05:50:28.181010       1 cleaner.go:81] Starting CSR cleaner controller
W0211 05:50:28.233620       1 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.

I0211 05:50:30.261842       1 taint_controller.go:158] Sending events to api server.
I0211 05:50:30.261999       1 controllermanager.go:493] Started "node"
I0211 05:50:30.262041       1 controllermanager.go:483] Starting "replicaset"
I0211 05:50:30.262241       1 node_controller.go:558] Starting node controller
I0211 05:50:30.262281       1 controller_utils.go:1019] Waiting for caches to sync for node controller
I0211 05:50:30.401807       1 controllermanager.go:493] Started "replicaset"
I0211 05:50:30.401915       1 controllermanager.go:483] Starting "horizontalpodautoscaling"
I0211 05:50:30.401983       1 replica_set.go:183] Starting replicaset controller
I0211 05:50:30.402016       1 controller_utils.go:1019] Waiting for caches to sync for ReplicaSet controller
I0211 05:50:31.115152       1 controllermanager.go:493] Started "horizontalpodautoscaling"
I0211 05:50:31.115190       1 controllermanager.go:483] Starting "disruption"
I0211 05:50:31.115282       1 horizontal.go:128] Starting HPA controller
I0211 05:50:31.115296       1 controller_utils.go:1019] Waiting for caches to sync for HPA controller
I0211 05:50:31.290236       1 garbagecollector.go:182] syncing garbage collector with updated resources from discovery: map[{extensions v1beta1 daemonsets}:{} {apps v1 replicasets}:{} {networking.k8s.io v1 networkpolicies}:{} { v1 configmaps}:{} { v1 persistentvolumes}:{} { v1 events}:{} {apps v1 daemonsets}:{} {apps v1 statefulsets}:{} { v1 endpoints}:{} {events.k8s.io v1beta1 events}:{} {batch v1beta1 cronjobs}:{} { v1 namespaces}:{} {extensions v1beta1 deployments}:{} {certificates.k8s.io v1beta1 certificatesigningrequests}:{} {rbac.authorization.k8s.io v1 rolebindings}:{} {storage.k8s.io v1 storageclasses}:{} {admissionregistration.k8s.io v1beta1 validatingwebhookconfigurations}:{} { v1 persistentvolumeclaims}:{} { v1 serviceaccounts}:{} {batch v1 jobs}:{} {rbac.authorization.k8s.io v1 clusterrolebindings}:{} {admissionregistration.k8s.io v1beta1 mutatingwebhookconfigurations}:{} {apiextensions.k8s.io v1beta1 customresourcedefinitions}:{} { v1 secrets}:{} { v1 pods}:{} { v1 podtemplates}:{} {extensions v1beta1 networkpolicies}:{} {apps v1 controllerrevisions}:{} {autoscaling v1 horizontalpodautoscalers}:{} {policy v1beta1 poddisruptionbudgets}:{} {rbac.authorization.k8s.io v1 roles}:{} { v1 limitranges}:{} {extensions v1beta1 podsecuritypolicies}:{} {extensions v1beta1 ingresses}:{} {apps v1 deployments}:{} {rbac.authorization.k8s.io v1 clusterroles}:{} { v1 resourcequotas}:{} {apiregistration.k8s.io v1beta1 apiservices}:{} {extensions v1beta1 replicasets}:{} { v1 services}:{} { v1 nodes}:{} { v1 replicationcontrollers}:{}]
I0211 05:50:31.367111       1 controllermanager.go:493] Started "disruption"
I0211 05:50:31.367133       1 controllermanager.go:483] Starting "statefulset"
I0211 05:50:31.367214       1 disruption.go:288] Starting disruption controller
I0211 05:50:31.370501       1 controller_utils.go:1019] Waiting for caches to sync for disruption controller
I0211 05:50:31.602543       1 controllermanager.go:493] Started "statefulset"
I0211 05:50:31.602566       1 controllermanager.go:483] Starting "csrsigning"
I0211 05:50:31.602604       1 stateful_set.go:150] Starting stateful set controller
I0211 05:50:31.602617       1 controller_utils.go:1019] Waiting for caches to sync for stateful set controller
I0211 05:50:31.755235       1 controllermanager.go:493] Started "csrsigning"
I0211 05:50:31.755286       1 controllermanager.go:483] Starting "endpoint"
I0211 05:50:31.755318       1 certificate_controller.go:113] Starting certificate controller
I0211 05:50:31.755357       1 controller_utils.go:1019] Waiting for caches to sync for certificate controller
I0211 05:50:32.002965       1 controllermanager.go:493] Started "endpoint"
I0211 05:50:32.003006       1 controllermanager.go:483] Starting "serviceaccount"
I0211 05:50:32.003041       1 endpoints_controller.go:153] Starting endpoint controller
I0211 05:50:32.003054       1 controller_utils.go:1019] Waiting for caches to sync for endpoint controller
I0211 05:50:32.251459       1 controllermanager.go:493] Started "serviceaccount"
I0211 05:50:32.251483       1 controllermanager.go:483] Starting "deployment"
I0211 05:50:32.251516       1 serviceaccounts_controller.go:115] Starting service account controller
I0211 05:50:32.251524       1 controller_utils.go:1019] Waiting for caches to sync for service account controller
I0211 05:50:32.501238       1 controllermanager.go:493] Started "deployment"
I0211 05:50:32.501790       1 deployment_controller.go:153] Starting deployment controller
I0211 05:50:32.501811       1 controller_utils.go:1019] Waiting for caches to sync for deployment controller
I0211 05:50:32.506632       1 controller_utils.go:1019] Waiting for caches to sync for resource quota controller
I0211 05:50:32.554465       1 controller_utils.go:1026] Caches are synced for ClusterRoleAggregator controller
I0211 05:50:32.554469       1 controller_utils.go:1026] Caches are synced for service account controller
I0211 05:50:32.558334       1 controller_utils.go:1026] Caches are synced for certificate controller
E0211 05:50:32.565143       1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="k8s-master-40117112-0"  does not exist
E0211 05:50:32.565201       1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="k8s-worker-40117112-0"  does not exist
I0211 05:50:32.582421       1 controller_utils.go:1026] Caches are synced for TTL controller
I0211 05:50:32.587448       1 controller_utils.go:1026] Caches are synced for ReplicationController controller
I0211 05:50:32.599266       1 controller_utils.go:1019] Waiting for caches to sync for garbage collector controller
I0211 05:50:32.601133       1 ttl_controller.go:271] Changed ttl annotation for node k8s-master-40117112-0 to 0 seconds
I0211 05:50:32.603411       1 controller_utils.go:1026] Caches are synced for endpoint controller
I0211 05:50:32.603479       1 controller_utils.go:1026] Caches are synced for deployment controller
I0211 05:50:32.603494       1 controller_utils.go:1026] Caches are synced for GC controller
I0211 05:50:32.603647       1 controller_utils.go:1026] Caches are synced for ReplicaSet controller
I0211 05:50:32.607359       1 ttl_controller.go:271] Changed ttl annotation for node k8s-worker-40117112-0 to 0 seconds
I0211 05:50:32.609699       1 controller_utils.go:1026] Caches are synced for certificate controller
I0211 05:50:32.629952       1 controller_utils.go:1026] Caches are synced for namespace controller
I0211 05:50:32.631223       1 controller_utils.go:1026] Caches are synced for HPA controller
I0211 05:50:32.643167       1 controller_utils.go:1026] Caches are synced for job controller
I0211 05:50:32.643223       1 controller_utils.go:1026] Caches are synced for service controller
I0211 05:50:32.643299       1 service_controller.go:654] Detected change in list of current cluster nodes. New node set: [k8s-master-40117112-0 k8s-worker-40117112-0]
I0211 05:50:32.643330       1 service_controller.go:662] Successfully updated 0 out of 0 load balancers to direct traffic to the updated set of nodes
I0211 05:50:32.671630       1 controller_utils.go:1026] Caches are synced for disruption controller
I0211 05:50:32.671723       1 disruption.go:296] Sending events to api server.
E0211 05:50:32.677658       1 clusterroleaggregation_controller.go:180] edit failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "edit": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:32.702997       1 controller_utils.go:1026] Caches are synced for stateful set controller
E0211 05:50:32.712337       1 clusterroleaggregation_controller.go:180] view failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "view": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:32.735146       1 controller_utils.go:1026] Caches are synced for attach detach controller
I0211 05:50:32.775163       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:32.775200       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:32.775491       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:32.775516       1 controller_utils.go:1026] Caches are synced for persistent volume controller
I0211 05:50:32.862427       1 controller_utils.go:1026] Caches are synced for node controller
I0211 05:50:32.862555       1 node_controller.go:633] Controller observed a new Node: "k8s-master-40117112-0"
I0211 05:50:32.862639       1 controller_utils.go:197] Recording Registered Node k8s-master-40117112-0 in Controller event message for node k8s-master-40117112-0
I0211 05:50:32.862717       1 node_controller.go:611] Initializing eviction metric for zone: southcentralus:�:0
I0211 05:50:32.862780       1 node_controller.go:633] Controller observed a new Node: "k8s-worker-40117112-0"
I0211 05:50:32.862805       1 controller_utils.go:197] Recording Registered Node k8s-worker-40117112-0 in Controller event message for node k8s-worker-40117112-0
W0211 05:50:32.862922       1 node_controller.go:964] Missing timestamp for Node k8s-master-40117112-0. Assuming now as a timestamp.
W0211 05:50:32.862964       1 node_controller.go:964] Missing timestamp for Node k8s-worker-40117112-0. Assuming now as a timestamp.
I0211 05:50:32.863026       1 node_controller.go:880] Controller detected that zone southcentralus:�:0 is now in state Normal.
I0211 05:50:32.863512       1 taint_controller.go:181] Starting NoExecuteTaintManager
I0211 05:50:32.863597       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"k8s-worker-40117112-0", UID:"73740a28-0eef-11e8-851a-000d3a752906", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node k8s-worker-40117112-0 event: Registered Node k8s-worker-40117112-0 in Controller
I0211 05:50:32.863648       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"k8s-master-40117112-0", UID:"7362eeec-0eef-11e8-851a-000d3a752906", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node k8s-master-40117112-0 event: Registered Node k8s-master-40117112-0 in Controller
I0211 05:50:32.885970       1 controller_utils.go:1026] Caches are synced for resource quota controller
I0211 05:50:32.893225       1 controller_utils.go:1026] Caches are synced for daemon sets controller
I0211 05:50:32.894095       1 controller_utils.go:1026] Caches are synced for garbage collector controller
I0211 05:50:32.894106       1 garbagecollector.go:144] Garbage collector: all resource monitors have synced.  Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
I0211 05:50:33.720047       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"303", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.720658       1 deployment_controller.go:485] Error syncing deployment kube-system/heapster: Operation cannot be fulfilled on deployments.extensions "heapster": the object has been modified; please apply your changes to the latest version and try again
E0211 05:50:33.723027       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.726213       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.743869       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.744003       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.744112       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.744558       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.754578       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.754628       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.754635       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.754667       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.760623       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.761024       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.760947       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.774945       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.784691       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.784928       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.784879       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.866815       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.883712       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.883860       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.883946       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.029379       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e7672b-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"323", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set tiller-deploy-589f6788d7 to 1
I0211 05:50:34.029878       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/tiller-deploy-589f6788d7, need 1, creating 1
I0211 05:50:34.039388       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:34.042697       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:34.042766       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:34.043505       1 deployment_controller.go:485] Error syncing deployment kube-system/tiller-deploy: Operation cannot be fulfilled on deployments.extensions "tiller-deploy": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:34.048950       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:34.053782       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"tiller-deploy-589f6788d7", UID:"79eb6d39-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"324", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: tiller-deploy-589f6788d7-x49tv
I0211 05:50:34.064608       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
I0211 05:50:34.065732       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
E0211 05:50:34.066504       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.390543       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:34.396889       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:34.396964       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.397157       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.535077       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:34.535237       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:34.535401       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:34.562823       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/kube-dns-v20-597689868c, need 2, creating 2
I0211 05:50:34.563868       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"kube-dns-v20", UID:"7a3a1ae2-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"349", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set kube-dns-v20-597689868c to 2
I0211 05:50:34.582518       1 deployment_controller.go:485] Error syncing deployment kube-system/kube-dns-v20: Operation cannot be fulfilled on deployments.extensions "kube-dns-v20": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:34.588342       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kube-dns-v20-597689868c", UID:"7a3c3d4f-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"351", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-dns-v20-597689868c-r4qsc
I0211 05:50:34.614439       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kube-dns-v20-597689868c", UID:"7a3c3d4f-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"351", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-dns-v20-597689868c-sjmxm
I0211 05:50:34.916450       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:34.916519       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:34.916875       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:34.991423       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"7a7a0ad9-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"375", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-85z7d
I0211 05:50:34.997729       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"7a7a0ad9-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"375", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-tpkc5
I0211 05:50:35.037121       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:35.042374       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: heapster-56b857ddcc-66zf9
I0211 05:50:35.775600       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:35.775707       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:35.775770       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:37.024208       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/kubernetes-dashboard-5dfcbcfb5c, need 1, creating 1
I0211 05:50:37.031076       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af736c4-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"405", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set kubernetes-dashboard-5dfcbcfb5c to 1
I0211 05:50:37.051202       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kubernetes-dashboard-5dfcbcfb5c", UID:"7bb46428-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"408", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kubernetes-dashboard-5dfcbcfb5c-fftnc
I0211 05:50:37.069942       1 deployment_controller.go:485] Error syncing deployment kube-system/kubernetes-dashboard: Operation cannot be fulfilled on deployments.extensions "kubernetes-dashboard": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:37.827710       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:37.828134       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 10s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:37.829436       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:39.092016       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:39.092074       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 10s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:39.092486       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:39.594505       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:39.594560       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 10s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:39.595051       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:39.971516       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:39.971580       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 10s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:39.972019       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:40.816779       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:40.816836       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 10s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:40.817408       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:47.867094       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:47.867265       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 20s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:47.867371       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:49.170888       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:49.171522       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 20s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:49.171715       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:49.636306       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:49.636379       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 20s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:49.636844       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:50.010733       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:50.010798       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 20s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:50.011233       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:50.862988       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:50.863058       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 20s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:50.863196       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:08.023961       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:08.024127       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 40s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:08.024278       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:09.213926       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:09.214180       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 40s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:09.214246       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:09.677321       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:09.677373       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 40s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:09.677409       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:10.048547       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:10.048604       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 40s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:51:10.048684       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:51:10.903084       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:10.903130       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 40s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:10.903164       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:48.062707       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:48.062835       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 1m20s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:48.063719       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:49.259820       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:49.260726       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 1m20s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:49.260986       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:49.719469       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:49.719640       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 1m20s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:49.720424       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:50.092085       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 05:51:50.092796       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
E0211 05:51:50.092245       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 1m20s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:51:50.939508       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:50.939569       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 1m20s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:50.939604       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:53.488947       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-77776c69b, need 1, creating 1
I0211 05:51:53.490226       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"heapster", UID:"79a8c0b0-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"609", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set heapster-77776c69b to 1
I0211 05:51:53.510165       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-77776c69b", UID:"a9484491-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"611", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: heapster-77776c69b-2b5h8
I0211 05:51:53.524115       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"heapster", UID:"79a8c0b0-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"609", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set heapster-56b857ddcc to 0
I0211 05:51:53.526860       1 replica_set.go:526] Too many replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 0, deleting 1
I0211 05:51:53.527648       1 controller_utils.go:590] Controller heapster-56b857ddcc deleting pod kube-system/heapster-56b857ddcc-66zf9
I0211 05:51:53.532783       1 deployment_controller.go:485] Error syncing deployment kube-system/heapster: Operation cannot be fulfilled on deployments.extensions "heapster": the object has been modified; please apply your changes to the latest version and try again
I0211 05:51:53.708372       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"617", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: heapster-56b857ddcc-66zf9
I0211 05:53:08.108215       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:08.108437       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 2m40s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:53:08.108519       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:53:09.301454       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:09.301537       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 2m40s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:53:09.301715       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:53:09.792969       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:09.793033       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 2m40s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:53:09.793100       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:53:10.135479       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:10.135672       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 2m40s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:53:10.135780       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:53:11.016962       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:11.017009       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 2m40s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:53:11.017139       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:55:48.543202       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:48.543251       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:55:48.543425       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:55:49.396169       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:49.396447       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:55:49.396531       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:55:49.829409       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:49.829555       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:55:49.829709       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:55:50.180718       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:50.180811       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:55:50.181459       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:55:51.061053       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:51.061111       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:55:51.061205       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:00:48.857764       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:48.857835       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:00:48.857925       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:00:49.436299       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:49.436360       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:00:49.436562       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:00:49.884395       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:49.884452       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:00:49.884521       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:00:50.218974       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:50.219028       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:00:50.219098       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:00:51.110232       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:51.110498       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:00:51.110644       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:05:49.026334       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:49.026393       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:05:49.026606       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:05:49.534528       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:49.534617       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:05:49.534724       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:05:49.919998       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:49.920045       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:05:49.920236       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:05:50.257611       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:50.257673       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:05:50.257781       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:05:51.165267       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:51.165311       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:05:51.165582       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found

After creating a service of type internal load balancer, these new log lines got printed
I0211 06:07:44.089297       1 service_controller.go:301] Ensuring LB for service default/ingress-nginx
I0211 06:07:44.089637       1 azure_loadbalancer.go:435] reconcileLoadBalancer(default/ingress-nginx) - wantLb(false): started
I0211 06:07:44.090674       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"ingress-nginx", UID:"dfe22760-0ef1-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"1861", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0211 06:07:44.228196       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 06:07:44.228224       1 azure_loadbalancer.go:441] reconcileLoadBalancer(default/ingress-nginx): lb(kube-5a7fd7a8) wantLb(false) resolved load balancer name
I0211 06:07:44.228241       1 azure_loadbalancer.go:797] ensure(default/ingress-nginx): lb(kube-5a7fd7a8) finished
I0211 06:07:44.285503       1 azure_backoff.go:248] PublicIPAddressesClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 06:07:44.285556       1 azure_loadbalancer.go:435] reconcileLoadBalancer(default/ingress-nginx) - wantLb(true): started
I0211 06:07:44.322257       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 06:07:44.322287       1 azure_loadbalancer.go:242] selectLoadBalancer: cluster(kube-5a7fd7a8) service(default/ingress-nginx) isInternal(true) - availabilitysetsnames [worker-availabilitySet-40117112]
I0211 06:07:44.322351       1 azure_loadbalancer.go:441] reconcileLoadBalancer(default/ingress-nginx): lb(kube-5a7fd7a8-internal) wantLb(true) resolved load balancer name
I0211 06:07:44.949151       1 azure_backoff.go:394] backoff: success, HTTP response=201
I0211 06:07:45.329576       1 azure_backoff.go:54] backoff: success
I0211 06:07:45.521068       1 azure_backoff.go:54] backoff: success
I0211 06:07:45.521115       1 azure_loadbalancer.go:797] ensure(default/ingress-nginx): lb(kube-5a7fd7a8-internal) finished
I0211 06:07:45.651638       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"ingress-nginx", UID:"dfe22760-0ef1-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"1861", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0211 06:10:49.216519       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:49.216651       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:10:49.216757       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:10:49.585439       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:49.585508       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:10:49.585588       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:10:49.962871       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:49.963071       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:10:49.963140       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:10:50.307581       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:50.313925       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:10:50.314360       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:10:51.215423       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:51.215571       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:10:51.215694       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found

kuberernetes server and cli version
kubectl version                                                                                [11:41]
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

acs-engine version
./acs-engine version
Version: canary
GitCommit: fe57150
GitTreeState: clean

Internal load balancer service which i tested
cat << EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app: ingress-nginx
EOF

acs-engine template used to create kubernetes cluster
{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.9",
      "orchestratorVersion": "1.9.3"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "kube",
      "vmSize": "Standard_B2s"
    },
    "agentPoolProfiles": [
      {
        "name": "worker",
        "count": 1,
        "vmSize": "Standard_B2s",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
      "secret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
  }
}

No target network IP configurations present for the internal load balancer even after 15 mins

If i create a public load balancer service, it's target network IP configurations are set automatically immediately

It seems that in 1.9.3 it's fixed - https://github.com/kubernetes/kubernetes/pull/59083, https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#changelog-since-v192

@kaluzaaa I also thought so, but it is not working still.
Can someone please confirm that they also hitting this problem too by creating the cluster v1.9.3 using acs-engine? Or point out at what step I might be doing wrong? I have provided the my acs-engine template.json and service.yaml above.

@diwakar-s-maurya Thanks of reporting the problem. There is another bug for internal load balancers. Will file a PR to fix it.

I ended up writing a powershell function to fix the internal load balancer.  Here it is in case someone is interested:
`
--------
function global:FixLoadBalancers($resourceGroup) {
# hacks here to get around bugs in the acs-engine loadbalancer code
Write-Host "Checking if load balancers are setup correctly for resourceGroup: $resourceGroup"
# 1. assign the nics to the loadbalancer
# find loadbalancer with name
$loadbalancer = "${resourceGroup}-internal"
$loadbalancerBackendPoolName = $resourceGroup # the name may change in the future so we should look it up
# for each worker VM
$virtualmachines = az vm list -g $resourceGroup --query "[].name" -o tsv
ForEach ($vm in $virtualmachines) {
    if ($vm -match "master" ) {}
    else {
        # for each worker VM
        Write-Host "Checking VM: $vm"
        # get first nic
        # $nic = "k8s-linuxagent-14964077-nic-0"
        $nicId = $(az vm nic list -g $resourceGroup --vm-name $vm --query "[].id" -o tsv)
        $nic = $(az network nic show --ids $nicId --resource-group $resourceGroup --query "name" -o tsv)

        # get first ipconfig of nic
        $ipconfig = $(az network nic ip-config list --resource-group $resourceGroup --nic-name $nic --query "[?primary].name" -o tsv)

        $loadbalancerForNic = $(az network nic ip-config show --resource-group $resourceGroup --nic-name $nic --name $ipconfig --query "loadBalancerBackendAddressPools[].id" -o tsv)
        # if loadBalancerBackendAddressPools is missing then
        if ([string]::IsNullOrEmpty($loadbalancerForNic)) {
            Write-Warning "Fixing load balancer for vm: $vm by adding nic $nic with ip-config $ipconfig to backend pool $loadbalancerBackendPoolName in load balancer $loadbalancer "
            # --lb-address-pools: Space-separated list of names or IDs of load balancer address pools to associate with the NIC. If names are used, --lb-name must be specified.
            az network nic ip-config update --resource-group $resourceGroup --nic-name $nic --name $ipconfig --lb-name $loadbalancer --lb-address-pools $loadbalancerBackendPoolName
        }
        elseif (!($($loadbalancerForNic -contains $loadbalancer))) {
            Write-Host "nic is already bound to load balancer $loadbalancerForNic with ip-config $ipconfig"
            Write-Host "adding internal load balancer to secondary ip-config"
            # get the first secondary ipconfig
            $ipconfig = $(az network nic ip-config list --resource-group $resourceGroup --nic-name $nic --query "[?!primary].name" -o tsv)[0]
            $loadbalancerForNic = $(az network nic ip-config show --resource-group $resourceGroup --nic-name $nic --name $ipconfig --query "loadBalancerBackendAddressPools[].id" -o tsv)
            if ([string]::IsNullOrEmpty($loadbalancerForNic)) {
                Write-Warning "Fixing load balancer for vm: $vm by adding nic $nic with ip-config $ipconfig to backend pool $loadbalancerBackendPoolName in load balancer $loadbalancer "
                # --lb-address-pools: Space-separated list of names or IDs of load balancer address pools to associate with the NIC. If names are used, --lb-name must be specified.
                az network nic ip-config update --resource-group $resourceGroup --nic-name $nic --name $ipconfig --lb-name $loadbalancer --lb-address-pools $loadbalancerBackendPoolName
            }
            else {
                Write-Host "Load Balancer with ip-config $ipconfig is already setup properly for vm: $vm"
            }
        }
        else {
            Write-Host "Load Balancer with ip-config $ipconfig is already setup properly for vm: $vm"
        }
    }
}

# 2. fix the ports in load balancing rules
Write-Host "Checking if the correct ports are setup in the load balancer"

# get frontendip configs for this IP
# $idToIPTuplesJson=$(az network lb frontend-ip list --resource-group=$AKS_PERS_RESOURCE_GROUP --lb-name $loadbalancer --query "[*].[id,privateIpAddress]")
# $idToIPTuplesJson = $(az network lb frontend-ip list --resource-group=$AKS_PERS_RESOURCE_GROUP --lb-name $loadbalancer --query "[*].{id:id,ip:privateIpAddress}")
$idToIPTuples = $(az network lb frontend-ip list --resource-group=$resourceGroup --lb-name $loadbalancer --query "[*].{id:id,ip:privateIpAddress}") | ConvertFrom-Json
$services = $($(kubectl get services --all-namespaces -o json) | ConvertFrom-Json).items
$loadBalancerServices = @()
Write-Host "---- Searching for kub services of type LoadBalancer"
foreach ($service in $services) {
    if ($($service.spec.type -eq "LoadBalancer")) {
        if($service.status.loadBalancer.ingress.Count -gt 0){
            Write-Host "Found kub services $($service.metadata.name) with $($service.status.loadBalancer.ingress[0].ip)"
            $loadBalancerServices += $service
        }
        else {
            Write-Host "Found kub services $($service.metadata.name) but it has no ingress IP so skipping it"
        }
    }
}
Write-Host "---- Finished searching for kub services of type LoadBalancer"

ForEach ($tuple in $idToIPTuples) {
    Write-Host "---------- tuple: $($tuple.ip)  $($tuple.id) ------------------"
    $rulesForIp = $(az network lb rule list --resource-group $resourceGroup --lb-name $loadbalancer --query "[?frontendIpConfiguration.id == '$($tuple.id)'].{frontid:frontendIpConfiguration.id,name:name,backendPort:backendPort,frontendPort: frontendPort}") | ConvertFrom-Json

    ForEach ($service in $loadBalancerServices) {
        Write-Host "-------- Checking kub service: $($service.metadata.name) ----"
        # first check ports for internal loadbalancer
        $loadBalancerIp = $($service.status.loadBalancer.ingress[0].ip)
        # Write-Host "Checking tuple ip $($tuple.ip) with loadBalancer Ip $loadBalancerIp"
        if ($tuple.ip -eq $loadBalancerIp) {
            #this is the right load balancer
            ForEach ($rule in $rulesForIp) {
                Write-Host "----- Checking rule $($rule.name) ----"
                # Write-Host "tuple $($tuple.ip) matches loadBalancerIP: $loadBalancerIp"
                # match rule.backendPort to $loadbalancerInfo.spec.ports
                ForEach ( $loadbalancerPortInfo in $($service.spec.ports)) {
                    # Write-Host "Rule: $rule "
                    # Write-Host "LoadBalancer:$loadbalancerPortInfo"
                    if ($($rule.frontendPort) -eq $($loadbalancerPortInfo.port)) {
                        Write-Host "Found matching frontend ports: rule: $($rule.frontendPort) of rule $($rule.name) and loadbalancer: $($loadbalancerPortInfo.port) from $($loadbalancerPortInfo.name)"
                        if ($($rule.backendPort) -ne $($loadbalancerPortInfo.nodePort)) {
                            Write-Warning "Backend ports don't match.  Will change $($rule.backendPort) to $($loadbalancerPortInfo.nodePort)"
                            # set the rule backendPort to nodePort instead
                            $rule.backendPort = $loadbalancerPortInfo.nodePort
                            az network lb rule update --lb-name $loadbalancer --name $($rule.name) --resource-group $resourceGroup --backend-port $loadbalancerPortInfo.nodePort
                        }
                        else {
                            Write-Host "Skipping changing backend port since it already matches $($rule.backendPort) vs $($loadbalancerPortInfo.nodePort)"
                        }
                    }
                    else {
                        Write-Host "Skipping rule $($rule.name): Rule port: $($rule.backendPort) is not a match for loadbalancerPort $($loadbalancerPortInfo.port) from $($loadbalancerPortInfo.name)"
                    }
                }
            }
            # get port from kubernetes service
        }
        else {
            Write-Host "Skipping tuple since tuple ip $($tuple.ip) does not match loadBalancerIP: $loadBalancerIp"
        }
    }
    Write-Host ""
}
# end hacks

}
`

Can confirm the bug for v1.9.3
Public LB is getting updated, internal LB is not updated.

@mblaschke The cherry-pick PR https://github.com/kubernetes/kubernetes/pull/59754 has been merged. It should be included in v1.9.4.

I just provisioned v 1.9.4 and I have this issue
~/r/j/cluster-manifests   master *…  kubectl describe svc kube-dns -n kube-system                                                             1078ms  Thu Mar 15 12:09:17 2018
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=KubeDNS
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kube-dns","kubernetes.io/cluster-service":"true","kubernetes.io/n...
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                <IP_ADDRESS>
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         <IP_ADDRESS>:53,<IP_ADDRESS>:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         <IP_ADDRESS>:53,<IP_ADDRESS>:53
Session Affinity:  None
Events:
Type     Reason                      Age                 From                Message

Warning  CreatingLoadBalancerFailed  4m (x212 over 17h)  service-controller  Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found

@JerryJoyce Since this is a ClusterIP service, the warning message could be omit safely. We will remove this warning in future releases.

@feiskyer is there an upstream issue/PR to address this behavior?

@feiskyer We are seeing additional symptoms that suggest that non-cloudprovider services are being processed via cloudprovider and being sent to Azure API. We're seeing Azure cloudprovider backoff entries in the logs which are generated from responses from Azure API.
The reason this would be bad is that add'l, unnecessary requests to Azure API count against per-subscription network quota. A large cluster that generated a significant number of these unnecessary request could send an entire subscription (and thus all of its clusters) into a network backoff state (no network requests allowed by Azure API).
Have you seen this? @lachie83

PR https://github.com/kubernetes/kubernetes/pull/61260 is for removing this warning for 1.9  release.

@feiskyer We are seeing additional symptoms that suggest that non-cloudprovider services are being processed via cloudprovider and being sent to Azure API. We're seeing Azure cloudprovider backoff entries in the logs which are generated from responses from Azure API.

The check is required because a service may be changed from LoadBalancer to ClusterIP. Cloud provider should ensure ALB cleaned up in this case.  In v1.10, caches for LB/NSG/RouteTable/VMSS/VM have been added, which could help to reduce such API calls.

@feiskyer I don't think your PR addresses these issues:
https://github.com/kubernetes/kubernetes/issues/61100
https://github.com/kubernetes/kubernetes/issues/60528

@jackfrancis This is working as expected. See @brendandburns's comments here.


Vocabulary

Hi,
It is written in your AAAI paper "To generate the answer vocabulary, we choose the top 1,000 most frequent answers in the train split as the answer vocabulary, which covers 84.7% / 86.2% / 85.6% of the train/val/test answers, respectively".
However, choosing the same vocabulary, I obtain significantly lower coverage ( about 78% for test and 80% for val). Do you have an idea why? Do you process further the answers?
Best,
Antoine Yang

Same questions. I got 84.7% / 82.2% / 78.6% for the train/val/test answers. Did you find the reason? Thx.


new version/tag needed after adding bower.json

In order to install via bower, a new tag (e.g. 0.3.3) is needed.

Believe various updates have been made to Bower installation. Please open new issue, if need be.


Gh 659 consistent edge creation

I have made the Edge class immutable (removed default constructors and setters). I've also done the same to the Entity class to have some consistency between the two Element implementations.
The classes are not fully immutable currently. In order to swap the source and destination in the Edge class I have kept the fields non-final. I have also left the "putIdentifier" methods on both classes. This is currently used only when creating ElementTuple objects, but it does allow a way of modifying the fields on an Element. If this is undesirable, then I'd suggest opening two new issues:

Move vertex swapping in the Edge class to Builder method. This will have the side effect of forcing the source and destination to be specified when invoking the build() method. Currently it's possible to create an empty Edge (i.e new Edge.Builder().build()), which has null for both source and destination.
Make ElementTuple immutable.

Apologies for the number of changes - this was always going to be a problem when changing the Edge and Entity classes.

Codecov Report

Merging #835 into develop will decrease coverage by 0.3%.
The diff coverage is 79%.

@@              Coverage Diff              @@
##             develop     #835      +/-   ##
=============================================
- Coverage      58.74%   58.44%   -0.31%
+ Complexity      2669     2621      -48
=============================================
  Files            489      486       -3
  Lines          12220    12142      -78
  Branches        1341     1344       +3
=============================================
- Hits            7179     7096      -83
- Misses          4453     4460       +7
+ Partials         588      586       -2

Impacted Files
Coverage Δ
Complexity Δ

...ava/uk/gov/gchq/gaffer/data/element/id/EdgeId.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

...a/uk/gov/gchq/gaffer/data/element/id/EntityId.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

...q/gaffer/rest/example/ExampleElementGenerator.java
9.09% <0%> (ø)
1 <0> (ø)
:arrow_down:

...hq/gaffer/mapstore/impl/GetAdjacentIdsHandler.java
83.05% <100%> (-0.29%)
4 <0> (ø)

.../export/resultcache/GafferResultCacheExporter.java
76.25% <100%> (ø)
7 <0> (ø)
:arrow_down:

...q/gaffer/spark/serialisation/kryo/Registrator.java
100% <100%> (ø)
2 <0> (ø)
:arrow_down:

.../gov/gchq/gaffer/rest/service/ExamplesService.java
63.38% <78.78%> (+2.55%)
28 <1> (ø)
:arrow_down:

...ain/java/uk/gov/gchq/gaffer/data/element/Edge.java
76.08% <85.71%> (-12.23%)
22 <10> (-2)

...n/java/uk/gov/gchq/gaffer/data/element/Entity.java
78.57% <92.85%> (-8.93%)
14 <3> (-2)

.../java/uk/gov/gchq/gaffer/data/element/Element.java
66.66% <0%> (-11.12%)
16% <0%> (-2%)

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50198c7...27fe697. Read the comment docs.

Looks good - there are a couple of minor comments above.
I'm going to raise an issue about providing an example in the documentation about how this change affects queries which have a filter that specifies the destination, e.g. if undirected edge B-A is inserted into a graph, and B is queried for, with a filter saying that the destination should be A or match A* then no results will be returned, because B is the destination. When executing such queries on undirected edges, the user should specify that either the source or the destination is A.

Closing the pull request while the issue with OPERATION_RETURN_MATCHED_SEEDS_AS_EDGE_SOURCE is investigated.


There should be a default image of the product on the product review page if no actual image for that product.

Bagisto Version: Master
Steps To reproduce--
1. Log in as an admin.
2. Create a configurable product.
3. Do not upload images for the parent product.
4. Visit the shop.
5. Try to review the created product.
Expected result: There should be a default image of the product on the product review page if no actual image for that product.
Actual result:  Getting the blank space if there is no image of parent product's in configurable products. Please find the screenshot Here

Hey, @Sanjaybhattwebkul  I can give it a go can you assign this issue to me?

please has this issue been solved?

@Sanjaybhattwebkul
Not able to reproduce the issue. Click Here