install kubernetes 1.14.0 using kube-aws won't work

Hi I am trying to upgrade kubernetes by setting:
kubernetesVersion: v1.14.0
in cluster.xml, but it won't work, and failed while building CF stack for controller.
Any suggestions if we can use kube-aws to build k8s cluster  other than the default version supported?
thanks
Li

@daili Hey! It is likely that 1.14 has some backward incompatible changes that preventing controller nodes from coming up.
To debug, would you mind sharing the output of kubelet and other system pods/containers(if exists) using journalctl and docker logs?

Hi @mumoshu thanks for your response
the journalctl give thousand of erros like below
-- Unit<EMAIL_ADDRESS>has finished shutting down.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd[1]: Removed slice User Slice of root.
-- Subject: Unit user-0.slice has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Unit user-0.slice has finished shutting down.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-timesyncd[750]: Timed out waiting for reply from <IP_ADDRESS>:123 (3.coreos.pool.ntp.org).
Mar 20 10:00:05 ip-10-11-128-202.mypc.local sudo[6164]:     core : TTY=pts/0 ; PWD=/tmp/packer-provisioner-ansible-local/5c9173b6-1cba-eafc-5605-45689b07e6c9 ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-icbnbkmutlqvrgtypaxxftrzim
mpbqox; /opt/active_python/bin/python /home/core/.ansible/tmp/ansible-tmp-1553036405.26-30996492961390/AnsiballZ_file.py
Mar 20 10:00:05 ip-10-11-128-202.mypc.local sudo[6164]: pam_unix(sudo:session): session opened for user root by core(uid=0)
Mar 20 10:00:05 ip-10-11-128-202.mypc.local sudo[6164]: pam_systemd(sudo:session): Cannot create session: Already running in a session
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: repeater ERROR: 2019/03/19 23:00:05 cloudwatch_journal_repeater.go:202: Error from putEvents : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: main ERROR: 2019/03/19 23:00:05 workers.go:48: Failed to write to cloudwatch batch size = : %d %s %v 0 failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: repeater ERROR: 2019/03/19 23:00:05 cloudwatch_journal_repeater.go:202: Error from putEvents : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: main ERROR: 2019/03/19 23:00:05 workers.go:48: Failed to write to cloudwatch batch size = : %d %s %v 0 failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: repeater ERROR: 2019/03/19 23:00:05 cloudwatch_journal_repeater.go:202: Error from putEvents : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:05 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: main ERROR: 2019/03/19 23:00:05 workers.go:48: Failed to write to cloudwatch batch size = : %d %s %v 0 failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.

Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd[1]: Starting Garbage Collection for rkt...
-- Subject: Unit rkt-gc.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Unit rkt-gc.service has begun starting up.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-timedated[6252]: Set NTP to enabled
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: repeater ERROR: 2019/03/19 23:00:06 cloudwatch_journal_repeater.go:202: Error from putEvents : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: main ERROR: 2019/03/19 23:00:06 workers.go:48: Failed to write to cloudwatch batch size = : %d %s %v 0 failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: repeater ERROR: 2019/03/19 23:00:06 cloudwatch_journal_repeater.go:202: Error from putEvents : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: main ERROR: 2019/03/19 23:00:06 workers.go:48: Failed to write to cloudwatch batch size = : %d %s %v 0 failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: repeater ERROR: 2019/03/19 23:00:06 cloudwatch_journal_repeater.go:202: Error from putEvents : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]: main ERROR: 2019/03/19 23:00:06 workers.go:48: Failed to write to cloudwatch batch size = : %d %s %v 0 failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors failed to put events: : NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors NoCredentialProviders: no valid providers in chain. Deprecated.
Mar 20 10:00:06 ip-10-11-128-202.mypc.local systemd-cloud-watch_linux[6135]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 20 10:00:06 ip-10-11-128-202.mypc.local sudo[6271]: pam_unix(sudo:session): session closed for user root

Hi @mumoshu
the controller pod log:
/var/log/pods/kube-system_kube-controller-manager-ip-10-11-69-69.ap-southeast-2.compute.internal_d792f2db94bd172d924d279407268cd9/kube-controller-manager
{"log":"Error: unknown flag: --insecure-experimental-approve-all-kubelet-csrs-for-group\n","stream":"stderr","time":"2019-04-05T01:26:03.938737816Z"}
{"log":"Usage:\n","stream":"stderr","time":"2019-04-05T01:26:03.939257488Z"}
{"log":"  hyperkube kube-controller-manager [flags]\n","stream":"stderr","time":"2019-04-05T01:26:03.93926912Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939273256Z"}
{"log":"Debugging flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939276712Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.93928021Z"}
{"log":"      --contention-profiling   Enable lock contention profiling, if profiling is enabled\n","stream":"stderr","time":"2019-04-05T01:26:03.939283663Z"}
{"log":"      --profiling              Enable profiling via web interface host:port/debug/pprof/\n","stream":"stderr","time":"2019-04-05T01:26:03.939287276Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939290807Z"}
{"log":"Generic flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939294126Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939297666Z"}
{"log":"      --allocate-node-cidrs                    Should CIDRs for Pods be allocated and set on the cloud provider.\n","stream":"stderr","time":"2019-04-05T01:26:03.939301016Z"}
{"log":"      --cidr-allocator-type string             Type of CIDR allocator to use (default "RangeAllocator")\n","stream":"stderr","time":"2019-04-05T01:26:03.939304688Z"}
{"log":"      --cloud-config string                    The path to the cloud provider configuration file. Empty string for no configuration file.\n","stream":"stderr","time":"2019-04-05T01:26:03.939308497Z"}
{"log":"      --cloud-provider string                  The provider for cloud services. Empty string for no provider.\n","stream":"stderr","time":"2019-04-05T01:26:03.939313298Z"}
{"log":"      --cluster-cidr string                    CIDR Range for Pods in cluster. Requires --allocate-node-cidrs to be true\n","stream":"stderr","time":"2019-04-05T01:26:03.939316921Z"}
{"log":"      --cluster-name string                    The instance prefix for the cluster. (default "kubernetes")\n","stream":"stderr","time":"2019-04-05T01:26:03.939320525Z"}
{"log":"      --configure-cloud-routes                 Should CIDRs allocated by allocate-node-cidrs be configured on the cloud provider. (default true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939324219Z"}
{"log":"      --controller-start-interval duration     Interval between starting controller managers.\n","stream":"stderr","time":"2019-04-05T01:26:03.939327974Z"}
{"log":"      --controllers strings                    A list of controllers to enable. '' enables all on-by-default controllers, 'foo' enables the controller named 'foo', '-foo' disables the controller named 'foo'.\n","stream":"stderr","time":"2019-04-05T01:26:03.939331604Z"}
{"log":"                                               All controllers: attachdetach, bootstrapsigner, cloud-node-lifecycle, clusterrole-aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, garbagecollector, horizontalpodautoscaling, job, namespace, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv-protection, pvc-protection, replicaset, replicationcontroller, resourcequota, root-ca-cert-publisher, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after-finished\n","stream":"stderr","time":"2019-04-05T01:26:03.939360784Z"}
{"log":"                                               Disabled-by-default controllers: bootstrapsigner, tokencleaner (default [])\n","stream":"stderr","time":"2019-04-05T01:26:03.939378283Z"}
{"log":"      --external-cloud-volume-plugin string    The plugin to use when cloud provider is set to external. Can be empty, should only be set when cloud-provider is external. Currently used to allow node and volume controllers to work for in tree cloud providers.\n","stream":"stderr","time":"2019-04-05T01:26:03.939382097Z"}
{"log":"      --feature-gates mapStringBool            A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:\n","stream":"stderr","time":"2019-04-05T01:26:03.939386133Z"}
{"log":"                                               APIListChunking=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939398622Z"}
{"log":"                                               APIResponseCompression=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.93940266Z"}
{"log":"                                               AllAlpha=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939406302Z"}
{"log":"                                               AppArmor=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939409967Z"}
{"log":"                                               AttachVolumeLimit=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.93941351Z"}
{"log":"                                               BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939417136Z"}
{"log":"                                               BlockVolume=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939421259Z"}
{"log":"                                               BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939424837Z"}
{"log":"                                               CPUManager=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939428462Z"}
{"log":"                                               CRIContainerLogRotation=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939432123Z"}
{"log":"                                               CSIBlockVolume=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939435699Z"}
{"log":"                                               CSIDriverRegistry=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939439289Z"}
{"log":"                                               CSIInlineVolume=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939442896Z"}
{"log":"                                               CSIMigration=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939446606Z"}
{"log":"                                               CSIMigrationAWS=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939450195Z"}
{"log":"                                               CSIMigrationGCE=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939453821Z"}
{"log":"                                               CSIMigrationOpenStack=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.93945791Z"}
{"log":"                                               CSINodeInfo=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939461718Z"}
{"log":"                                               CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939465353Z"}
{"log":"                                               CustomResourcePublishOpenAPI=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939468947Z"}
{"log":"                                               CustomResourceSubresources=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939472555Z"}
{"log":"                                               CustomResourceValidation=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939476238Z"}
{"log":"                                               CustomResourceWebhookConversion=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.93947977Z"}
{"log":"                                               DebugContainers=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939486391Z"}
{"log":"                                               DevicePlugins=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939490013Z"}
{"log":"                                               DryRun=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939493561Z"}
{"log":"                                               DynamicAuditing=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939497097Z"}
{"log":"                                               DynamicKubeletConfig=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939500629Z"}
{"log":"                                               ExpandCSIVolumes=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939516441Z"}
{"log":"                                               ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.93952131Z"}
{"log":"                                               ExpandPersistentVolumes=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939525486Z"}
{"log":"                                               ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939528875Z"}
{"log":"                                               ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939532609Z"}
{"log":"                                               HyperVContainer=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939536275Z"}
{"log":"                                               KubeletPodResources=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939539855Z"}
{"log":"                                               LocalStorageCapacityIsolation=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939543963Z"}
{"log":"                                               MountContainers=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939547645Z"}
{"log":"                                               NodeLease=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939551295Z"}
{"log":"                                               PodShareProcessNamespace=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939554799Z"}
{"log":"                                               ProcMountType=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939558365Z"}
{"log":"                                               QOSReserved=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939561919Z"}
{"log":"                                               ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939565442Z"}
{"log":"                                               ResourceQuotaScopeSelectors=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.93956901Z"}
{"log":"                                               RotateKubeletClientCertificate=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939572658Z"}
{"log":"                                               RotateKubeletServerCertificate=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939576216Z"}
{"log":"                                               RunAsGroup=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939579778Z"}
{"log":"                                               RuntimeClass=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939583372Z"}
{"log":"                                               SCTPSupport=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939590077Z"}
{"log":"                                               ScheduleDaemonSetPods=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939593689Z"}
{"log":"                                               ServerSideApply=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939597281Z"}
{"log":"                                               ServiceNodeExclusion=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939600852Z"}
{"log":"                                               StorageVersionHash=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.93960449Z"}
{"log":"                                               StreamingProxyRedirects=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939608065Z"}
{"log":"                                               SupportNodePidsLimit=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939611641Z"}
{"log":"                                               SupportPodPidsLimit=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939615282Z"}
{"log":"                                               Sysctls=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939618868Z"}
{"log":"                                               TTLAfterFinished=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939622734Z"}
{"log":"                                               TaintBasedEvictions=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939626485Z"}
{"log":"                                               TaintNodesByCondition=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939630153Z"}
{"log":"                                               TokenRequest=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939633705Z"}
{"log":"                                               TokenRequestProjection=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939637267Z"}
{"log":"                                               ValidateProxyRedirects=true|false (BETA - default=true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939640837Z"}
{"log":"                                               VolumeSnapshotDataSource=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939644485Z"}
{"log":"                                               VolumeSubpathEnvExpansion=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939648119Z"}
{"log":"                                               WinDSR=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939652645Z"}
{"log":"                                               WinOverlay=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939655491Z"}
{"log":"                                               WindowsGMSA=true|false (ALPHA - default=false)\n","stream":"stderr","time":"2019-04-05T01:26:03.939659276Z"}
{"log":"      --kube-api-burst int32                   Burst to use while talking with kubernetes apiserver. (default 30)\n","stream":"stderr","time":"2019-04-05T01:26:03.939662909Z"}
{"log":"      --kube-api-content-type string           Content type of requests sent to apiserver. (default "application/vnd.kubernetes.protobuf")\n","stream":"stderr","time":"2019-04-05T01:26:03.939684129Z"}
{"log":"      --kube-api-qps float32                   QPS to use while talking with kubernetes apiserver. (default 20)\n","stream":"stderr","time":"2019-04-05T01:26:03.939688094Z"}
{"log":"      --leader-elect                           Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability. (default true)\n","stream":"stderr","time":"2019-04-05T01:26:03.93969175Z"}
{"log":"      --leader-elect-lease-duration duration   The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled. (default 15s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939698846Z"}
{"log":"      --leader-elect-renew-deadline duration   The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled. (default 10s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939703649Z"}
{"log":"      --leader-elect-resource-lock endpoints   The type of resource object that is used for locking during leader election. Supported options are endpoints (default) and configmaps. (default "endpoints")\n","stream":"stderr","time":"2019-04-05T01:26:03.939707591Z"}
{"log":"      --leader-elect-retry-period duration     The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled. (default 2s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939711529Z"}
{"log":"      --min-resync-period duration             The resync period in reflectors will be random between MinResyncPeriod and 2MinResyncPeriod. (default 12h0m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939715343Z"}
{"log":"      --node-monitor-period duration           The period for syncing NodeStatus in NodeController. (default 5s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939719094Z"}
{"log":"      --route-reconciliation-period duration   The period for reconciling routes created for Nodes by cloud provider. (default 10s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939722667Z"}
{"log":"      --use-service-account-credentials        If true, use individual service account credentials for each controller.\n","stream":"stderr","time":"2019-04-05T01:26:03.939726292Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939729923Z"}
{"log":"Service controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939733361Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939736796Z"}
{"log":"      --concurrent-service-syncs int32   The number of services that are allowed to sync concurrently. Larger number = more responsive service management, but more CPU (and network) load (default 1)\n","stream":"stderr","time":"2019-04-05T01:26:03.939740152Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939743917Z"}
{"log":"Secure serving flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.93974734Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939750741Z"}
{"log":"      --bind-address ip                        The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (<IP_ADDRESS> for all IPv4 interfaces and :: for all IPv6 interfaces). (default <IP_ADDRESS>)\n","stream":"stderr","time":"2019-04-05T01:26:03.939754115Z"}
{"log":"      --cert-dir string                        The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.\n","stream":"stderr","time":"2019-04-05T01:26:03.939758306Z"}
{"log":"      --http2-max-streams-per-connection int   The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.\n","stream":"stderr","time":"2019-04-05T01:26:03.939762311Z"}
{"log":"      --secure-port int                        The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. (default 10257)\n","stream":"stderr","time":"2019-04-05T01:26:03.939766012Z"}
{"log":"      --tls-cert-file string                   File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.\n","stream":"stderr","time":"2019-04-05T01:26:03.939774167Z"}
{"log":"      --tls-cipher-suites strings              Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use.  Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA\n","stream":"stderr","time":"2019-04-05T01:26:03.939779881Z"}
{"log":"      --tls-min-version string                 Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12\n","stream":"stderr","time":"2019-04-05T01:26:03.939785469Z"}
{"log":"      --tls-private-key-file string            File containing the default x509 private key matching --tls-cert-file.\n","stream":"stderr","time":"2019-04-05T01:26:03.939789121Z"}
{"log":"      --tls-sni-cert-key namedCertKey          A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:.foo.com,foo.com". (default [])\n","stream":"stderr","time":"2019-04-05T01:26:03.939793385Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939798252Z"}
{"log":"Insecure serving flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939801689Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939805158Z"}
{"log":"      --address ip   The IP address on which to serve the insecure --port (set to <IP_ADDRESS> for all IPv4 interfaces and :: for all IPv6 interfaces). (default <IP_ADDRESS>) (DEPRECATED: see --bind-address instead.)\n","stream":"stderr","time":"2019-04-05T01:26:03.939808588Z"}
{"log":"      --port int     The port on which to serve unsecured, unauthenticated access. Set to 0 to disable. (default 10252) (DEPRECATED: see --secure-port instead.)\n","stream":"stderr","time":"2019-04-05T01:26:03.939812467Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939816148Z"}
{"log":"Authentication flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939819525Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939823227Z"}
{"log":"      --authentication-kubeconfig string                  kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no client CA is looked up in the cluster.\n","stream":"stderr","time":"2019-04-05T01:26:03.939826699Z"}
{"log":"      --authentication-skip-lookup                        If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.\n","stream":"stderr","time":"2019-04-05T01:26:03.939830739Z"}
{"log":"      --authentication-token-webhook-cache-ttl duration   The duration to cache responses from the webhook token authenticator. (default 10s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939837365Z"}
{"log":"      --authentication-tolerate-lookup-failure            If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.\n","stream":"stderr","time":"2019-04-05T01:26:03.939841451Z"}
{"log":"      --client-ca-file string                             If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.\n","stream":"stderr","time":"2019-04-05T01:26:03.939845409Z"}
{"log":"      --requestheader-allowed-names strings               List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.\n","stream":"stderr","time":"2019-04-05T01:26:03.939849742Z"}
{"log":"      --requestheader-client-ca-file string               Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.\n","stream":"stderr","time":"2019-04-05T01:26:03.939853843Z"}
{"log":"      --requestheader-extra-headers-prefix strings        List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])\n","stream":"stderr","time":"2019-04-05T01:26:03.939857945Z"}
{"log":"      --requestheader-group-headers strings               List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])\n","stream":"stderr","time":"2019-04-05T01:26:03.939861621Z"}
{"log":"      --requestheader-username-headers strings            List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])\n","stream":"stderr","time":"2019-04-05T01:26:03.939865327Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939869029Z"}
{"log":"Authorization flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939872416Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939875839Z"}
{"log":"      --authorization-always-allow-paths strings                A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server. (default [/healthz])\n","stream":"stderr","time":"2019-04-05T01:26:03.939879167Z"}
{"log":"      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped by authorization are forbidden.\n","stream":"stderr","time":"2019-04-05T01:26:03.939883082Z"}
{"log":"      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939887114Z"}
{"log":"      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939890758Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939894434Z"}
{"log":"Attachdetach controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.93989778Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.93990132Z"}
{"log":"      --attach-detach-reconcile-sync-period duration   The reconciler sync wait time between volume attach detach. This duration must be larger than one second, and increasing this value from the default may allow for volumes to be mismatched with pods. (default 1m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939904897Z"}
{"log":"      --disable-attach-detach-reconcile-sync           Disable volume attach detach reconciler sync. Disabling this may cause volumes to be mismatched with pods. Use wisely.\n","stream":"stderr","time":"2019-04-05T01:26:03.939911518Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.93991527Z"}
{"log":"Csrsigning controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939918722Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939922206Z"}
{"log":"      --cluster-signing-cert-file string                 Filename containing a PEM-encoded X509 CA certificate used to issue cluster-scoped certificates (default "/etc/kubernetes/ca/ca.pem")\n","stream":"stderr","time":"2019-04-05T01:26:03.939925602Z"}
{"log":"      --cluster-signing-key-file string                  Filename containing a PEM-encoded RSA or ECDSA private key used to sign cluster-scoped certificates (default "/etc/kubernetes/ca/ca.key")\n","stream":"stderr","time":"2019-04-05T01:26:03.939929545Z"}
{"log":"      --experimental-cluster-signing-duration duration   The length of duration signed certificates will be given. (default 8760h0m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939933454Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939937127Z"}
{"log":"Deployment controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939940511Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939943946Z"}
{"log":"      --concurrent-deployment-syncs int32            The number of deployment objects that are allowed to sync concurrently. Larger number = more responsive deployments, but more CPU (and network) load (default 5)\n","stream":"stderr","time":"2019-04-05T01:26:03.939947276Z"}
{"log":"      --deployment-controller-sync-period duration   Period for syncing the deployments. (default 30s)\n","stream":"stderr","time":"2019-04-05T01:26:03.939951081Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939954646Z"}
{"log":"Deprecated flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939957966Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939961391Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939964671Z"}
{"log":"Endpoint controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939967978Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939971367Z"}
{"log":"      --concurrent-endpoint-syncs int32   The number of endpoint syncing operations that will be done concurrently. Larger number = faster endpoint updating, but more CPU (and network) load (default 5)\n","stream":"stderr","time":"2019-04-05T01:26:03.939974742Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939978474Z"}
{"log":"Garbagecollector controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939981802Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939985224Z"}
{"log":"      --concurrent-gc-syncs int32   The number of garbage collector workers that are allowed to sync concurrently. (default 20)\n","stream":"stderr","time":"2019-04-05T01:26:03.939988565Z"}
{"log":"      --enable-garbage-collector    Enables the generic garbage collector. MUST be synced with the corresponding flag of the kube-apiserver. (default true)\n","stream":"stderr","time":"2019-04-05T01:26:03.939992185Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.939995802Z"}
{"log":"Horizontalpodautoscaling controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.939999175Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940003458Z"}
{"log":"      --horizontal-pod-autoscaler-cpu-initialization-period duration   The period after pod start when CPU samples might be skipped. (default 5m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940006053Z"}
{"log":"      --horizontal-pod-autoscaler-downscale-stabilization duration     The period for which autoscaler will look backwards and not scale down below any recommendation it made during that period. (default 5m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940010092Z"}
{"log":"      --horizontal-pod-autoscaler-initial-readiness-delay duration     The period after pod start during which readiness changes will be treated as initial readiness. (default 30s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940016412Z"}
{"log":"      --horizontal-pod-autoscaler-sync-period duration                 The period for syncing the number of pods in horizontal pod autoscaler. (default 15s)\n","stream":"stderr","time":"2019-04-05T01:26:03.94002027Z"}
{"log":"      --horizontal-pod-autoscaler-tolerance float                      The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling. (default 0.1)\n","stream":"stderr","time":"2019-04-05T01:26:03.94002398Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940027747Z"}
{"log":"Namespace controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940031083Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940034555Z"}
{"log":"      --concurrent-namespace-syncs int32   The number of namespace objects that are allowed to sync concurrently. Larger number = more responsive namespace termination, but more CPU (and network) load (default 10)\n","stream":"stderr","time":"2019-04-05T01:26:03.940038654Z"}
{"log":"      --namespace-sync-period duration     The period for syncing namespace life-cycle updates (default 5m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940042432Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940046019Z"}
{"log":"Nodeipam controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940049356Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940052823Z"}
{"log":"      --node-cidr-mask-size int32         Mask size for node cidr in cluster. (default 24)\n","stream":"stderr","time":"2019-04-05T01:26:03.940056162Z"}
{"log":"      --service-cluster-ip-range string   CIDR Range for Services in cluster. Requires --allocate-node-cidrs to be true\n","stream":"stderr","time":"2019-04-05T01:26:03.94005974Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940063362Z"}
{"log":"Nodelifecycle controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940066705Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940070172Z"}
{"log":"      --enable-taint-manager                   WARNING: Beta feature. If set to true enables NoExecute Taints and will evict all not-tolerating Pod running on Nodes tainted with this kind of Taints. (default true)\n","stream":"stderr","time":"2019-04-05T01:26:03.940073515Z"}
{"log":"      --large-cluster-size-threshold int32     Number of nodes from which NodeController treats the cluster as large for the eviction logic purposes. --secondary-node-eviction-rate is implicitly overridden to 0 for clusters this size or smaller. (default 50)\n","stream":"stderr","time":"2019-04-05T01:26:03.940077343Z"}
{"log":"      --node-eviction-rate float32             Number of nodes per second on which pods are deleted in case of node failure when a zone is healthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters. (default 0.1)\n","stream":"stderr","time":"2019-04-05T01:26:03.940081295Z"}
{"log":"      --node-monitor-grace-period duration     Amount of time which we allow running Node to be unresponsive before marking it unhealthy. Must be N times more than kubelet's nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet to post node status. (default 40s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940085543Z"}
{"log":"      --node-startup-grace-period duration     Amount of time which we allow starting Node to be unresponsive before marking it unhealthy. (default 1m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940089524Z"}
{"log":"      --pod-eviction-timeout duration          The grace period for deleting pods on failed nodes. (default 5m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940093235Z"}
{"log":"      --secondary-node-eviction-rate float32   Number of nodes per second on which pods are deleted in case of node failure when a zone is unhealthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters. This value is implicitly overridden to 0 if the cluster size is smaller than --large-cluster-size-threshold. (default 0.01)\n","stream":"stderr","time":"2019-04-05T01:26:03.940099322Z"}
{"log":"      --unhealthy-zone-threshold float32       Fraction of Nodes in a zone which needs to be not Ready (minimum 3) for zone to be treated as unhealthy.  (default 0.55)\n","stream":"stderr","time":"2019-04-05T01:26:03.940103514Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940107181Z"}
{"log":"Persistentvolume-binder controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940110502Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940113937Z"}
{"log":"      --enable-dynamic-provisioning                         Enable dynamic provisioning for environments that support it. (default true)\n","stream":"stderr","time":"2019-04-05T01:26:03.940117259Z"}
{"log":"      --enable-hostpath-provisioner                         Enable HostPath PV provisioning when running without a cloud provider. This allows testing and development of provisioning features.  HostPath provisioning is not supported in any way, won't work in a multi-node cluster, and should not be used for anything other than testing or development.\n","stream":"stderr","time":"2019-04-05T01:26:03.940120907Z"}
{"log":"      --flex-volume-plugin-dir string                       Full path of the directory in which the flex volume plugin should search for additional third party volume plugins. (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")\n","stream":"stderr","time":"2019-04-05T01:26:03.940124995Z"}
{"log":"      --pv-recycler-increment-timeout-nfs int32             the increment of time added per Gi to ActiveDeadlineSeconds for an NFS scrubber pod (default 30)\n","stream":"stderr","time":"2019-04-05T01:26:03.940128948Z"}
{"log":"      --pv-recycler-minimum-timeout-hostpath int32          The minimum ActiveDeadlineSeconds to use for a HostPath Recycler pod.  This is for development and testing only and will not work in a multi-node cluster. (default 60)\n","stream":"stderr","time":"2019-04-05T01:26:03.940132987Z"}
{"log":"      --pv-recycler-minimum-timeout-nfs int32               The minimum ActiveDeadlineSeconds to use for an NFS Recycler pod (default 300)\n","stream":"stderr","time":"2019-04-05T01:26:03.940136799Z"}
{"log":"      --pv-recycler-pod-template-filepath-hostpath string   The file path to a pod definition used as a template for HostPath persistent volume recycling. This is for development and testing only and will not work in a multi-node cluster.\n","stream":"stderr","time":"2019-04-05T01:26:03.940140437Z"}
{"log":"      --pv-recycler-pod-template-filepath-nfs string        The file path to a pod definition used as a template for NFS persistent volume recycling\n","stream":"stderr","time":"2019-04-05T01:26:03.940144294Z"}
{"log":"      --pv-recycler-timeout-increment-hostpath int32        the increment of time added per Gi to ActiveDeadlineSeconds for a HostPath scrubber pod.  This is for development and testing only and will not work in a multi-node cluster. (default 30)\n","stream":"stderr","time":"2019-04-05T01:26:03.940147957Z"}
{"log":"      --pvclaimbinder-sync-period duration                  The period for syncing persistent volumes and persistent volume claims (default 15s)\n","stream":"stderr","time":"2019-04-05T01:26:03.9401518Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940155514Z"}
{"log":"Podgc controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.94015885Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940162291Z"}
{"log":"      --terminated-pod-gc-threshold int32   Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If \u003c= 0, the terminated pod garbage collector is disabled. (default 12500)\n","stream":"stderr","time":"2019-04-05T01:26:03.940165665Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940172298Z"}
{"log":"Replicaset controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940175959Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940179538Z"}
{"log":"      --concurrent-replicaset-syncs int32   The number of replica sets that are allowed to sync concurrently. Larger number = more responsive replica management, but more CPU (and network) load (default 5)\n","stream":"stderr","time":"2019-04-05T01:26:03.940182879Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940186623Z"}
{"log":"Replicationcontroller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.94018995Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940193392Z"}
{"log":"      --concurrent_rc_syncs int32   The number of replication controllers that are allowed to sync concurrently. Larger number = more responsive replica management, but more CPU (and network) load (default 5)\n","stream":"stderr","time":"2019-04-05T01:26:03.94019675Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940200517Z"}
{"log":"Resourcequota controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940203841Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.94020728Z"}
{"log":"      --concurrent-resource-quota-syncs int32   The number of resource quotas that are allowed to sync concurrently. Larger number = more responsive quota management, but more CPU (and network) load (default 5)\n","stream":"stderr","time":"2019-04-05T01:26:03.940210932Z"}
{"log":"      --resource-quota-sync-period duration     The period for syncing quota usage status in the system (default 5m0s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940214814Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940218456Z"}
{"log":"Serviceaccount controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940221846Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940225299Z"}
{"log":"      --concurrent-serviceaccount-token-syncs int32   The number of service account token objects that are allowed to sync concurrently. Larger number = more responsive token generation, but more CPU (and network) load (default 5)\n","stream":"stderr","time":"2019-04-05T01:26:03.940228631Z"}
{"log":"      --root-ca-file string                           If set, this root certificate authority will be included in service account's token secret. This must be a valid PEM-encoded CA bundle.\n","stream":"stderr","time":"2019-04-05T01:26:03.940232519Z"}
{"log":"      --service-account-private-key-file string       Filename containing a PEM-encoded private RSA or ECDSA key used to sign service account tokens.\n","stream":"stderr","time":"2019-04-05T01:26:03.940236344Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940240058Z"}
{"log":"Ttl-after-finished controller flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940243386Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940246848Z"}
{"log":"      --concurrent-ttl-after-finished-syncs int32   The number of TTL-after-finished controller workers that are allowed to sync concurrently. (default 5)\n","stream":"stderr","time":"2019-04-05T01:26:03.940250199Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940253856Z"}
{"log":"Misc flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.94025721Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940260599Z"}
{"log":"      --kubeconfig string   Path to kubeconfig file with authorization and master location information.\n","stream":"stderr","time":"2019-04-05T01:26:03.940263948Z"}
{"log":"      --master string       The address of the Kubernetes API server (overrides any value in kubeconfig).\n","stream":"stderr","time":"2019-04-05T01:26:03.940267571Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940273453Z"}
{"log":"Global flags:\n","stream":"stderr","time":"2019-04-05T01:26:03.940276865Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.94028023Z"}
{"log":"      --alsologtostderr                  log to standard error as well as files\n","stream":"stderr","time":"2019-04-05T01:26:03.940283594Z"}
{"log":"  -h, --help                             help for kube-controller-manager\n","stream":"stderr","time":"2019-04-05T01:26:03.940287193Z"}
{"log":"      --log-backtrace-at traceLocation   when logging hits line file:N, emit a stack trace (default :0)\n","stream":"stderr","time":"2019-04-05T01:26:03.940290711Z"}
{"log":"      --log-dir string                   If non-empty, write log files in this directory\n","stream":"stderr","time":"2019-04-05T01:26:03.94029425Z"}
{"log":"      --log-file string                  If non-empty, use this log file\n","stream":"stderr","time":"2019-04-05T01:26:03.940297825Z"}
{"log":"      --log-flush-frequency duration     Maximum number of seconds between log flushes (default 5s)\n","stream":"stderr","time":"2019-04-05T01:26:03.940301327Z"}
{"log":"      --logtostderr                      log to standard error instead of files (default true)\n","stream":"stderr","time":"2019-04-05T01:26:03.940304872Z"}
{"log":"      --skip-headers                     If true, avoid header prefixes in the log messages\n","stream":"stderr","time":"2019-04-05T01:26:03.940308424Z"}
{"log":"      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)\n","stream":"stderr","time":"2019-04-05T01:26:03.94031198Z"}
{"log":"  -v, --v Level                          number for the log level verbosity\n","stream":"stderr","time":"2019-04-05T01:26:03.940315892Z"}
{"log":"      --version version[=true]           Print version information and quit\n","stream":"stderr","time":"2019-04-05T01:26:03.940319414Z"}
{"log":"      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging\n","stream":"stderr","time":"2019-04-05T01:26:03.940322893Z"}
{"log":"\n","stream":"stderr","time":"2019-04-05T01:26:03.940326449Z"}
{"log":"unknown flag: --insecure-experimental-approve-all-kubelet-csrs-for-group\n","stream":"stderr","time":"2019-04-05T01:26:03.940329784Z"}
Now the api pod log:
"log":"E0405 00:59:44.206998       1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted\n","stream":"stderr","time":"2019-04-05T00:59:44.207049593Z"}
lrwxrwxrwx. 1 root root  165 Apr  5 12:20 9.log -> /var/lib/docker/containers/cf186684f2420251ec42c78d254e31e010d3771bf6f352148fa79efc92df892f/cf186684f2420251ec42c78d254e31e010d3771bf6f352148fa79efc92df892f-json.log
{"log":"W0405 00:59:41.255757       1 feature_gate.go:218] Setting GA feature gate PodPriority=true. It will be removed in a future release.\n","stream":"stderr","time":"2019-04-05T00:59:41.25593388Z"}
{"log":"I0405 00:59:41.255953       1 server.go:559] external host was not specified, using <IP_ADDRESS>\n","stream":"stderr","time":"2019-04-05T00:59:41.269971434Z"}
{"log":"I0405 00:59:41.256086       1 server.go:146] Version: v1.14.0\n","stream":"stderr","time":"2019-04-05T00:59:41.269987911Z"}
{"log":"W0405 00:59:41.797177       1 admission.go:76] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.\n","stream":"stderr","time":"2019-04-05T00:59:41.797303493Z"}
{"log":"I0405 00:59:41.797881       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook.\n","stream":"stderr","time":"2019-04-05T00:59:41.79794916Z"}
{"log":"I0405 00:59:41.797988       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.\n","stream":"stderr","time":"2019-04-05T00:59:41.798015655Z"}
{"log":"W0405 00:59:41.798409       1 admission.go:76] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.\n","stream":"stderr","time":"2019-04-05T00:59:41.798471722Z"}
{"log":"E0405 00:59:41.798888       1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted\n","stream":"stderr","time":"2019-04-05T00:59:41.798946029Z"}
{"log":"E0405 00:59:41.799027       1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted\n","stream":"stderr","time":"2019-04-05T00:59:41.799055645Z"}
{"log":"E0405 00:59:41.799118       1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted\n","stream":"stderr","time":"2019-04-05T00:59:41.79917431Z"}
{"log":"E0405 00:59:41.799278       1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted\n","stream":"stderr","time":"2019-04-05T00:59:41.79930706Z"}
{"log":"E0405 00:59:41.799384       1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted\n","stream":"stderr","time":"2019-04-05T00:59:41.799416137Z"}
{"log":"E0405 00:59:41.799477       1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted\n","stream":"stderr","time":"2019-04-05T00:59:41.799503847Z"}
{"log":"I0405 00:59:41.799537       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook.\n","stream":"stderr","time":"2019-04-05T00:59:41.799563542Z"}
{"log":"I0405 00:59:41.799626       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.\n","stream":"stderr","time":"2019-04-05T00:59:41.79966035Z"}
{"log":"I0405 00:59:41.801525       1 client.go:352] parsed scheme: ""\n","stream":"stderr","time":"2019-04-05T00:59:41.801566668Z"}
{"log":"I0405 00:59:41.801611       1 client.go:352] scheme "" not registered, fallback to default scheme\n","stream":"stderr","time":"2019-04-05T00:59:41.801661094Z"}
{"log":"I0405 00:59:41.801794       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 0  \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.801826065Z"}
{"log":"I0405 00:59:41.801941       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.801973349Z"}
{"log":"I0405 00:59:41.820585       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.820675655Z"}
{"log":"I0405 00:59:41.821169       1 client.go:352] parsed scheme: ""\n","stream":"stderr","time":"2019-04-05T00:59:41.821210785Z"}
{"log":"I0405 00:59:41.821278       1 client.go:352] scheme "" not registered, fallback to default scheme\n","stream":"stderr","time":"2019-04-05T00:59:41.821305316Z"}
{"log":"I0405 00:59:41.821359       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 0  \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.821387533Z"}
{"log":"I0405 00:59:41.821487       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.821516291Z"}
{"log":"I0405 00:59:41.840179       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.840254822Z"}
{"log":"I0405 00:59:41.915477       1 master.go:233] Using reconciler: lease\n","stream":"stderr","time":"2019-04-05T00:59:41.915594416Z"}
{"log":"I0405 00:59:41.916080       1 client.go:352] parsed scheme: ""\n","stream":"stderr","time":"2019-04-05T00:59:41.916131885Z"}
{"log":"I0405 00:59:41.916173       1 client.go:352] scheme "" not registered, fallback to default scheme\n","stream":"stderr","time":"2019-04-05T00:59:41.916225589Z"}
{"log":"I0405 00:59:41.916312       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 0  \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.916341434Z"}
{"log":"I0405 00:59:41.916443       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.916489057Z"}
{"log":"I0405 00:59:41.957613       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{etcd0.etcd-cici-k8s-dev1.cici.itctest.my-aws.local:2379 \u003cnil\u003e}]\n","stream":"stderr","time":"2019-04-05T00:59:41.9577334Z"}
{"log":"I0405 00:59:41.959509       1 client.go:352] parsed scheme: ""\n","stream":"stderr","time":"2019-04-05T00:59:41.95956654Z"}
{"log":"I0405 00:59:41.959806       1 client.go:352] scheme "" not registered, fallback to default scheme\n","stream":"stderr","time":"2019-04-05T00:59:41.959852528Z"}
Now the kube-system_kube-scheduler pod log:
{"log":"W0405 00:59:41.564044       1 feature_gate.go:218] Setting GA feature gate PodPriority=true. It will be removed in a future release.\n","stream":"stderr","time":"2019-04-05T00:59:41.564189945Z"}
{"log":"I0405 00:59:42.232529       1 serving.go:319] Generated self-signed cert in-memory\n","stream":"stderr","time":"2019-04-05T00:59:42.232628366Z"}
{"log":"W0405 00:59:42.779787       1 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.\n","stream":"stderr","time":"2019-04-05T00:59:42.779887809Z"}
{"log":"W0405 00:59:42.779809       1 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.\n","stream":"stderr","time":"2019-04-05T00:59:42.779909039Z"}
{"log":"W0405 00:59:42.779822       1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.\n","stream":"stderr","time":"2019-04-05T00:59:42.779964073Z"}
{"log":"I0405 00:59:42.785371       1 server.go:142] Version: v1.14.0\n","stream":"stderr","time":"2019-04-05T00:59:42.785450086Z"}
{"log":"I0405 00:59:42.785422       1 defaults.go:87] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory\n","stream":"stderr","time":"2019-04-05T00:59:42.785498751Z"}
{"log":"W0405 00:59:42.786261       1 authorization.go:47] Authorization is disabled\n","stream":"stderr","time":"2019-04-05T00:59:42.786322467Z"}
{"log":"W0405 00:59:42.786281       1 authentication.go:55] Authentication is disabled\n","stream":"stderr","time":"2019-04-05T00:59:42.786334682Z"}
{"log":"I0405 00:59:42.786289       1 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251\n","stream":"stderr","time":"2019-04-05T00:59:42.786371321Z"}
{"log":"I0405 00:59:42.786740       1 secure_serving.go:116] Serving securely on [::]:10259\n","stream":"stderr","time":"2019-04-05T00:59:42.786807459Z"}
{"log":"E0405 00:59:45.937336       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.937434549Z"}
{"log":"E0405 00:59:45.937528       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.937806773Z"}
{"log":"E0405 00:59:45.937610       1 reflector.go:126] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:223: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.937879017Z"}
{"log":"E0405 00:59:45.937649       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.937929515Z"}
{"log":"E0405 00:59:45.937680       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.938006489Z"}
{"log":"E0405 00:59:45.937734       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.938056197Z"}
{"log":"E0405 00:59:45.938805       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.938867367Z"}
{"log":"E0405 00:59:45.938928       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.938978202Z"}
{"log":"E0405 00:59:45.939031       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.939077841Z"}
{"log":"E0405 00:59:45.945767       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:45.945832167Z"}
{"log":"E0405 00:59:46.938870       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.939013424Z"}
{"log":"E0405 00:59:46.940825       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.940894788Z"}
{"log":"E0405 00:59:46.941640       1 reflector.go:126] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:223: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.943115685Z"}
{"log":"E0405 00:59:46.944210       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.945305942Z"}
{"log":"E0405 00:59:46.946799       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.947390835Z"}
{"log":"E0405 00:59:46.948141       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.948241988Z"}
{"log":"E0405 00:59:46.948600       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.948693483Z"}
{"log":"E0405 00:59:46.950568       1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope\n","stream":"stderr","time":"2019-04-05T00:59:46.951520333Z"}

when i tried to curl the etcd server url on port 2379 I also got SSl error

was it relevant to https://github.com/kubernetes/kubernetes/pull/70144?

@daili Hey! Ithought etcd managed by kube-aws has mutual auth enabled since long time ago.
Could you share your cluster.yaml?
Your controller logs seem to indicate that there's authentication/authorization issue between apiserver and other k8s system components?

Sure @mumoshu

clusterName: cici-k8s-dev1
s3URI: s3://cici-kube-np-itc-myco-au
disableContainerLinuxAutomaticUpdates: true
sshAccessAllowedSourceCIDRs:

<IP_ADDRESS>/8
<IP_ADDRESS>/16
<IP_ADDRESS>/20
<IP_ADDRESS>/14
<IP_ADDRESS>/18
<IP_ADDRESS>/16
<IP_ADDRESS>/20
<IP_ADDRESS>/18
<IP_ADDRESS>/14

amiId: ami-079403c39816a39c8
apiEndpoints:

name: default
dnsName: api-cici-k8s-dev1.cici.itctest.my-aws.local
loadBalancer:
subnets:
- name: np-application-a
- name: np-application-b
- name: np-application-c
private: true
hostedZone:
id: Z2PA4OEZCHI6F6

keyName: itc-apro-dev1
region: ap-southeast-2
kmsKeyArn: arn:aws:kms:ap-southeast-2:123456789012:key/6f39e293-179f-4afd-bd30-acab02a1c3b4
controller:
instanceTags:
instanceRole: controller
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
instanceType: t2.large
autoScalingGroup:
minSize: 1
maxSize: 1
rollingUpdateMinInstancesInService: 1
iam:
instanceProfile:
arn: "arn:aws:iam::123456789012:instance-profile/kubernetesCiciInstanceProfileController"
subnets:
- name: np-application-a
- name: np-application-b
- name: np-application-c
rootVolume:
# Disk size (GiB) for worker nodes
size: 50
# Disk type for worker node (one of standard, io1, or gp2)
type: gp2
# Number of I/O operations per second (IOPS) that the worker node disk supports. Leave blank if worker.rootVolume.type is not io1
#iops: 0
worker:
nodePools:
# nodepool for application subnet a
- name: nodes-2a
subnets:
- name: np-application-a
iam:
instanceProfile:
arn: "arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
instanceType: m4.xlarge
autoScalingGroup:
minSize: 1
maxSize: 1
rollingUpdateMinInstancesInService: 1
awsNodeLabels:
enabled: false
instanceTags:
instanceRole: worker
kubernetes.io/autoscaler/cici-k8s-dev1: true
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
loadBalancer:
type: network
rootVolume:
# Disk size (GiB) for worker nodes
size: 50
# Disk type for worker node (one of standard, io1, or gp2)
type: gp2
# Number of I/O operations per second (IOPS) that the worker node disk supports. Leave blank if worker.rootVolume.type is not io1
#iops: 0
autoscaling:
clusterAutoscaler:
enabled: true
clusterAutoscalerSupport:
enabled: true
featureGates:
PodPriority: true
# nodepool for application subnet b
- name: nodes-2b
  subnets:
    - name: np-application-b
  iam:
    instanceProfile:
      arn: "arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
  instanceType: m4.xlarge
  autoScalingGroup:
    minSize: 1
    maxSize: 1
    rollingUpdateMinInstancesInService: 1
  awsNodeLabels:
    enabled: false
  instanceTags:
    instanceRole: worker
    kubernetes.io/autoscaler/cici-k8s-dev1: true
    'my:focalPoint:sdgPlatformCode': cici
    'state:scheduleName': StopAtEndOfSydneyBusinessHours
    'info:taggingVersion': 2
    'state:patching:mode': replace
    'my:focalPoint:sdgCode': 097
    'my:focalPoint:applicationId': 1024
    'my:env:subcode': dev1
    'my:env:code': np
  loadBalancer:
    type: network
  rootVolume:
    # Disk size (GiB) for worker nodes
    size: 50
    # Disk type for worker node (one of standard, io1, or gp2)
    type: gp2
    # Number of I/O operations per second (IOPS) that the worker node disk supports. Leave blank if worker.rootVolume.type is not io1
    #iops: 0
  autoscaling:
    clusterAutoscaler:
      enabled: true
  clusterAutoscalerSupport:
    enabled: true
  featureGates:
    PodPriority: true

# nodepool for application subnet c
- name: nodes-2c
  subnets:
    - name: np-application-c
  iam:
    instanceProfile:
      arn: "arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
  instanceType: m4.xlarge
  autoScalingGroup:
    minSize: 1
    maxSize: 1
    rollingUpdateMinInstancesInService: 1
  awsNodeLabels:
    enabled: false
  instanceTags:
    instanceRole: worker
    kubernetes.io/autoscaler/cici-k8s-dev1: true
    'my:focalPoint:sdgPlatformCode': cici
    'state:scheduleName': StopAtEndOfSydneyBusinessHours
    'info:taggingVersion': 2
    'state:patching:mode': replace
    'my:focalPoint:sdgCode': 097
    'my:focalPoint:applicationId': 1024
    'my:env:subcode': dev1
    'my:env:code': np
  loadBalancer:
    type: network
  rootVolume:
    # Disk size (GiB) for worker nodes
    size: 50
    # Disk type for worker node (one of standard, io1, or gp2)
    type: gp2
    # Number of I/O operations per second (IOPS) that the worker node disk supports. Leave blank if worker.rootVolume.type is not io1
    #iops: 0
  autoscaling:
    clusterAutoscaler:
      enabled: true
  clusterAutoscalerSupport:
    enabled: true
  featureGates:
    PodPriority: true

etcd:
count: 1
version: 3.3.10
instanceType: t2.large
instanceTags:
instanceRole: etcd
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
dataVolume:
encrypted: true
subnets:
- name: np-application-a
- name: np-application-b
- name: np-application-c
iam:
instanceProfile:
arn: "arn:aws:iam::123456789012:instance-profile/kubernetesCiciInstanceProfileEtcd"
memberIdentityProvider: eni
internalDomainName: "etcd-cici-k8s-dev1.cici.itctest.my-aws.local"
manageRecordSets: true
hostedZone:
id: "ABCDEDFH012"
rootVolume:
# Disk size (GiB) for worker nodes
size: 50
# Disk type for worker node (one of standard, io1, or gp2)
type: gp2
# Number of I/O operations per second (IOPS) that the worker node disk supports.  Recommended when rbac is enabled.
auditLog:
enabled: false
maxage: 30
logpath: /dev/stdout
See https://kubernetes.io/docs/admin/authentication/#webhook-token-authentication for more information
authentication:
webhook:
enabled: false
cacheTTL: 1m0s
configBase64: base64-encoded-webhook-yaml
Add predefined set of labels to the controller nodes
The set includes names of launch configurations and autoscaling groups
awsNodeLabels:
enabled: false
tlsBootstrap:
enabled: true
nodeAuthorizer:
enabled: true
ephemeralImageStorage:
enabled: false
kube2IamSupport:
enabled: false
When enabled, kubectl drain is run when the instance is being replaced by the auto scaling group, or when
the instance receives a termination notice (in case of spot instances)
nodeDrainer:
enabled: true
# Maximum time to wait, in minutes, for the node to be completely drained. Must be an integer between 1 and 60.
drainTimeout: 5
# IAM role to assume with kube2iam for the pod in "kube-node-drainer-asg-status-updater" deployment.
iamRole:
# Empty, inactive by default. Set it to valid ARN "arn: arn:aws:iam::0123456789012:role/roleName" to activate.
arn: "arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
Configure OpenID Connect token authenticator plugin in Kubernetes API server.
For using Dex as a custom OIDC provider, please check "contrib/dex/README.md".
WARNING: always use "https" for "issuerUrl", otherwise the Kubernetes API server will not start correctly.
oidc:
enabled: false
issuerUrl: "https://login.cici-id.com"
clientId: "cici-k8s-stg"
usernameClaim: "email"
groupsClaim: "groups"
disableSecurityGroupIngress: false
kubelet:
RotateCerts:
enabled: true

Thanks for sharing!
Unfortunately I see no prob in your config.
Are you sure you're using correct certs and keys for apiserver,
controller-manager and creating the cluster from scratch?
2019年4月8日(月) 8:08 daili<EMAIL_ADDRESS>
Sure @mumoshu https://github.com/mumoshu
clusterName: cici-k8s-dev1
s3URI: s3://cici-kube-np-itc-myco-au
disableContainerLinuxAutomaticUpdates: true
sshAccessAllowedSourceCIDRs:

<IP_ADDRESS>/8
<IP_ADDRESS>/16
<IP_ADDRESS>/20
<IP_ADDRESS>/14
<IP_ADDRESS>/18
<IP_ADDRESS>/16
<IP_ADDRESS>/20
<IP_ADDRESS>/18
<IP_ADDRESS>/14

amiId: ami-079403c39816a39c8
apiEndpoints:

name: default
dnsName: api-cici-k8s-dev1.cici.itctest.my-aws.local
loadBalancer:
subnets:

name: np-application-a
name: np-application-b
name: np-application-c
private: true
hostedZone:
id: Z2PA4OEZCHI6F6

keyName: itc-apro-dev1
region: ap-southeast-2
kmsKeyArn:
arn:aws:kms:ap-southeast-2:123456789012:key/6f39e293-179f-4afd-bd30-acab02a1c3b4
controller:
instanceTags:
instanceRole: controller
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
instanceType: t2.large
autoScalingGroup:
minSize: 1
maxSize: 1
rollingUpdateMinInstancesInService: 1
iam:
instanceProfile:
arn:
"arn:aws:iam::123456789012:instance-profile/kubernetesCiciInstanceProfileController"
subnets:

name: np-application-a
name: np-application-b
name: np-application-c
rootVolume:

Disk size (GiB) for worker nodes
size: 50
Disk type for worker node (one of standard, io1, or gp2)
type: gp2
Number of I/O operations per second (IOPS) that the worker node disk
supports. Leave blank if worker.rootVolume.type is not io1
#iops: 0
worker:
nodePools:
nodepool for application subnet a

name: nodes-2a
subnets:
name: np-application-a
iam:
instanceProfile:
arn:
"arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
instanceType: m4.xlarge
autoScalingGroup:
minSize: 1
maxSize: 1
rollingUpdateMinInstancesInService: 1
awsNodeLabels:
enabled: false
instanceTags:
instanceRole: worker
kubernetes.io/autoscaler/cici-k8s-dev1: true
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
loadBalancer:
type: network
rootVolume:

Disk size (GiB) for worker nodes
size: 50
Disk type for worker node (one of standard, io1, or gp2)
type: gp2
Number of I/O operations per second (IOPS) that the worker node disk
supports. Leave blank if worker.rootVolume.type is not io1
#iops: 0
autoscaling:
clusterAutoscaler:
enabled: true
clusterAutoscalerSupport:
enabled: true
featureGates:
PodPriority: true
nodepool for application subnet b

name: nodes-2b
subnets:

name: np-application-b
iam:
instanceProfile:
arn: "arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
instanceType: m4.xlarge
autoScalingGroup:
minSize: 1
maxSize: 1
rollingUpdateMinInstancesInService: 1
awsNodeLabels:
enabled: false
instanceTags:
instanceRole: worker
kubernetes.io/autoscaler/cici-k8s-dev1: true
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
loadBalancer:
type: network
rootVolume:

Disk size (GiB) for worker nodes
size: 50
Disk type for worker node (one of standard, io1, or gp2)
type: gp2
Number of I/O operations per second (IOPS) that the worker node disk supports. Leave blank if worker.rootVolume.type is not io1
#iops: 0
autoscaling:
clusterAutoscaler:
enabled: true
clusterAutoscalerSupport:
enabled: true
featureGates:
PodPriority: true

nodepool for application subnet c

name: nodes-2c
subnets:

name: np-application-c
iam:
instanceProfile:
arn: "arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
instanceType: m4.xlarge
autoScalingGroup:
minSize: 1
maxSize: 1
rollingUpdateMinInstancesInService: 1
awsNodeLabels:
enabled: false
instanceTags:
instanceRole: worker
kubernetes.io/autoscaler/cici-k8s-dev1: true
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
loadBalancer:
type: network
rootVolume:

Disk size (GiB) for worker nodes
size: 50
Disk type for worker node (one of standard, io1, or gp2)
type: gp2
Number of I/O operations per second (IOPS) that the worker node disk supports. Leave blank if worker.rootVolume.type is not io1
#iops: 0
autoscaling:
clusterAutoscaler:
enabled: true
clusterAutoscalerSupport:
enabled: true
featureGates:
PodPriority: true

etcd:
count: 1
version: 3.3.10
instanceType: t2.large
instanceTags:
instanceRole: etcd
'my:focalPoint:sdgPlatformCode': cici
'state:scheduleName': StopAtEndOfSydneyBusinessHours
'info:taggingVersion': 2
'state:patching:mode': replace
'my:focalPoint:sdgCode': 097
'my:focalPoint:applicationId': 1024
'my:env:subcode': dev1
'my:env:code': np
dataVolume:
encrypted: true
subnets:

name: np-application-a
name: np-application-b
name: np-application-c
iam:
instanceProfile:
arn:
"arn:aws:iam::123456789012:instance-profile/kubernetesCiciInstanceProfileEtcd"

memberIdentityProvider: eni
internalDomainName: "etcd-cici-k8s-dev1.cici.itctest.my-aws.local"
manageRecordSets: true
hostedZone:
id: "ABCDEDFH012"
rootVolume:
Disk size (GiB) for worker nodes
size: 50
Disk type for worker node (one of standard, io1, or gp2)
type: gp2
Number of I/O operations per second (IOPS) that the worker node disk
supports.  Recommended when rbac is enabled.
auditLog:
enabled: false
maxage: 30
logpath: /dev/stdout
See
https://kubernetes.io/docs/admin/authentication/#webhook-token-authentication
for more information
authentication:
webhook:
enabled: false
cacheTTL: 1m0s
configBase64: base64-encoded-webhook-yaml
Add predefined set of labels to the controller nodes The set includes
names of launch configurations and autoscaling groups
awsNodeLabels:
enabled: false
tlsBootstrap:
enabled: true
nodeAuthorizer:
enabled: true
ephemeralImageStorage:
enabled: false
kube2IamSupport:
enabled: false
When enabled, kubectl drain is run when the instance is being replaced by
the auto scaling group, or when the instance receives a termination
notice (in case of spot instances)
nodeDrainer:
enabled: true
Maximum time to wait, in minutes, for the node to be completely drained.
Must be an integer between 1 and 60.
drainTimeout: 5
IAM role to assume with kube2iam for the pod in
"kube-node-drainer-asg-status-updater" deployment.
iamRole:
Empty, inactive by default. Set it to valid ARN "arn:
arn:aws:iam::0123456789012:role/roleName" to activate.
arn:
"arn:aws:iam::123456789012:instance-profile/app-roles-iam-CiciKubeInstanceProfileWorker-1ABCDEFGHIJKLM"
Configure OpenID Connect token authenticator plugin in Kubernetes API
server. For using Dex as a custom OIDC provider, please check
"contrib/dex/README.md". WARNING: always use "https" for "issuerUrl",
otherwise the Kubernetes API server will not start correctly.
oidc:
enabled: false
issuerUrl: "https://login.cici-id.com"
clientId: "cici-k8s-stg"
usernameClaim: "email"
groupsClaim: "groups"
disableSecurityGroupIngress: false
kubelet:
RotateCerts:
enabled: true
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/kube-aws/issues/1578#issuecomment-480639407,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABV-VgB8C9HOswVRYRTKQ7PuDEizgraks5venpzgaJpZM4cb93y
.

@mumoshu  thanks for your prompt response. One quick question, do you know if kube-aws 0.12.0 has been tested against kubernetes 1.14.0?  When I check the doc, https://github.com/kubernetes-incubator/kube-aws/releases, it is said the kubernetes version is 1.11.3, so I was wondering  if 1.14.0 has never been tested although there is an argument in the cluster to support specification of kubernetes version.

I have not yet tested it with 1.14!
2019年4月8日(月) 10:37 daili<EMAIL_ADDRESS>
@mumoshu https://github.com/mumoshu thanks for your prompt response.
One quick question, do you know if kube-aws 0.12.0 has been tested against
kubernetes 1.14.0? When I check the doc,
https://github.com/kubernetes-incubator/kube-aws/releases, it is said the
kubernetes version is 1.11.3, so I was wondering if 1.14.0 has never been
tested although there is an argument in the cluster to support
specification of kubernetes version.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/kube-aws/issues/1578#issuecomment-480654378,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABV-QL3bLH-NPYWW31kudBK1H_m2Q_Aks5vep1AgaJpZM4cb93y
.

Hi @mumoshu  In The latest try , the outstanding error from journalctl is
Apr 08 13:13:05 ip-10-11-68-53.aws.mydc.local sh[1384]: W0408 03:13:05.752082    1384 cni.go:213] Unable to update cni config: No networks found in /etc/kubernetes/cni/net.d
Apr 08 13:13:05 ip-10-11-68-53.aws.mydc.local sh[1384]: E0408 03:13:05.768511    1384 pod_workers.go:190] Error syncing pod d792f2db94bd172d924d279407268cd9 ("kube-controller-manager-ip-10-11-68-53.ap-southeast-2.compute.internal_kube-system(d792f2db94bd172d924d279407268cd9)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-10-11-68-53.ap-southeast-2.compute.internal_kube-system(d792f2db94bd172d924d279407268cd9)"
Apr 08 13:13:06 ip-10-11-68-53.aws.mydc.local sh[1384]: E0408 03:13:06.050475    1384 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 08 13:13:10 ip-10-11-68-53.aws.mydc.local sh[1384]: W0408 03:13:10.752350    1384 cni.go:213] Unable to update cni config: No networks found in /etc/kubernetes/cni/net.d
Apr 08 13:13:11 ip-10-11-68-53.aws.mydc.local sh[1384]: E0408 03:13:11.051580    1384 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 08 13:13:15 ip-10-11-68-53.aws.mydc.local sh[1384]: W0408 03:13:15.752630    1384 cni.go:213] Unable to update cni config: No networks found in /etc/kubernetes/cni/net.d
Apr 08 13:13:16 ip-10-11-68-53.aws.mydc.local sh[1384]: E0408 03:13:16.052730    1384 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 08 13:13:17 ip-10-11-68-53.aws.mydc.local sh[1384]: E0408 03:13:17.768735    1384 pod_workers.go:190] Error syncing pod d792f2db94bd172d924d279407268cd9 ("kube-controller-manager-ip-10-11-68-53.ap-southeast-2.compute.internal_kube-system(d792f2db94bd172d924d279407268cd9)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-10-11-68-53.ap-southeast-2.compute.internal_kube-system(d792f2db94bd172d924d279407268cd9)"
Apr 08 13:13:20 ip-10-11-68-53.aws.mydc.local sh[1384]: W0408 03:13:20.752914    1384 cni.go:213] Unable to update cni config: No networks found in /etc/kubernetes/cni/net.d
Apr 08 13:13:21 ip-10-11-68-53.aws.mydc.local sh[1384]: E0408 03:13:21.058142    1384 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 08 13:13:25 ip-10-11-68-53.aws.mydc.local sh[1384]: W0408 03:13:25.753188    1384 cni.go:213] Unable to update cni config: No networks found in /etc/kubernetes/cni/net.d
Apr 08 13:13:26 ip-10-11-68-53.aws.mydc.local sh[1384]: E0408 03:13:26.059551    1384 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 08 13:13:27 ip-10-11-68-53.aws.mydc.local systemd[1]: cfn-signal.service: Start-pre operation timed out. Terminating.
Apr 08 13:13:27 ip-10-11-68-53.aws.mydc.local systemd[1]: cfn-signal.service: Failed with result 'timeout'.
Apr 08 13:13:27 ip-10-11-68-53.aws.mydc.local systemd[1]: Failed to start cfn-signal.service.

No matter it is canal or flannel network settings, it is all having this error. Any clue on it?

I think the feature introduced in 1.13.0
https://github.com/kubernetes/kubeadm/issues/1285
Is the thing that prevents kube-aws to work with 1.13.0 or later version.
Now to talk to the API server,  like shown from the test code:
https://github.com/kubernetes/kubernetes/blob/1588af4031e746e3e48855256054133e8707f018/test/integration/serving/serving_test.go#L133
more arguments are required.

I0409 01:39:14.746942       1 serving.go:318] Generated self-signed cert in-memory
W0409 01:39:15.199955       1 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W0409 01:39:15.199978       1 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W0409 01:39:15.199991       1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I0409 01:39:15.206077       1 server.go:150] Version: v1.13.4
I0409 01:39:15.206124       1 defaults.go:210] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
W0409 01:39:15.213012       1 authorization.go:47] Authorization is disabled
W0409 01:39:15.213029       1 authentication.go:55] Authentication is disabled
I0409 01:39:15.213038       1 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251
I0409 01:39:15.213419       1 secure_serving.go:116] Serving securely on [::]:10259
E0409 01:39:22.108201       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0409 01:39:22.108353       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0409 01:39:22.108376       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0409 01:39:22.108423       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0409 01:39:22.108507       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0409 01:39:22.108546       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0409 01:39:22.108662       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0409 01:39:22.115728       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0409 01:39:22.156929       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0409 01:39:22.157978       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0409 01:39:23.109205       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0409 01:39:23.111349       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0409 01:39:23.121953       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0409 01:39:23.127342       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0409 01:39:23.129462       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0409 01:39:23.139659       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0409 01:39:23.146578       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0409 01:39:23.151304       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0409 01:39:23.158705       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0409 01:39:23.159459       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
I0409 01:39:25.015244       1 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
I0409 01:39:25.118607       1 controller_utils.go:1034] Caches are synced for scheduler controller
I0409 01:39:25.118704       1 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-scheduler...
I0409 01:39:25.125098       1 leaderelection.go:214] successfully acquired lease kube-system/kube-scheduler

above is the try from install  v1.13.4 kubernetes.
I think the log is indicating issues mentioned here
https://github.com/kubernetes/kubeadm/issues/1285
where extra arguments are required to access the APIs but yet specified in current kube-aws generated code ?

@daili You found it!
That seems like an issue.
According to the decriptions of the flags, perhaps we need to pass these kubeconfig files to kube-scheduler and kube-controller-manager(or even better, create separate kubeconfig files for authorization and authentication? but that's an another story):
https://github.com/kubernetes-incubator/kube-aws/blob/e8acbeac55b7f7987767370cd321c91900bb4b85/builtin/files/userdata/cloud-config-controller#L4641-L4681

I have not tested it myself yet, but for controller-manager, you'll need to add another lines here:
https://github.com/kubernetes-incubator/kube-aws/blob/e8acbeac55b7f7987767370cd321c91900bb4b85/builtin/files/userdata/cloud-config-controller#L3259
So that it looks like:
- --kubeconfig=/etc/kubernetes/kubeconfig/kube-controller-manager.yaml
- --authentication-kubeconfig=/etc/kubernetes/kubeconfig/kube-controller-manager.yaml
- --authorization-kubeconfig=/etc/kubernetes/kubeconfig/kube-controller-manager.yaml

For kube-scheduler, https://github.com/kubernetes-incubator/kube-aws/blob/e8acbeac55b7f7987767370cd321c91900bb4b85/builtin/files/userdata/cloud-config-controller#L3354

No need to fork kube-aws to test it. Just edit cloud-config-controller rendered by kube-aws

@mumoshu thanks a lot,I will give it a try.

@mumoshu
The test result is like below, the argument issue errors seems gone, but new issue coming up
I0409 03:37:49.881793       1 serving.go:318] Generated self-signed cert in-memory
W0409 03:37:57.482039       1 authentication.go:262] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0409 03:37:57.482064       1 authentication.go:199] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0409 03:37:57.482075       1 authentication.go:200] Continuing without authentication configuration. This may treat all requests as anonymous.
W0409 03:37:57.482081       1 authentication.go:201] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0409 03:37:57.488518       1 server.go:150] Version: v1.13.4
I0409 03:37:57.488554       1 defaults.go:210] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
W0409 03:37:57.489406       1 authorization.go:47] Authorization is disabled
W0409 03:37:57.489491       1 authentication.go:55] Authentication is disabled
I0409 03:37:57.489571       1 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251
I0409 03:37:57.490241       1 secure_serving.go:116] Serving securely on [::]:10259
E0409 03:37:57.504080       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0409 03:37:57.504278       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0409 03:37:57.504434       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0409 03:37:57.504577       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0409 03:37:57.504688       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0409 03:37:57.504816       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0409 03:37:57.504964       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0409 03:37:57.505097       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0409 03:37:57.505177       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0409 03:37:57.505909       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0409 03:37:58.505765       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0409 03:37:58.506917       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0409 03:37:58.543676       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcont

Any clue on  that?

we need a kubeconfig file for metric-server to use with kube-scheduler authentication?

@daili Not sure why metrics-server was mentioned in the referenced kubeadm issue, but my understanding is that you need kubeconfigs so that scheduler and controller-manager is able to obtain tokens.
Then those tokens are used by scheduler and controller-manager to authenticate against K8S API

@mumoshu
for scheduler userdata, I tried both

  - path: /etc/kubernetes/manifests/kube-scheduler.yaml
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-scheduler
        namespace: kube-system
        labels:
          k8s-app: kube-scheduler
      spec:
        hostNetwork: true
        containers:
        - name: kube-scheduler
          image: {{.HyperkubeImage.RepoWithTag}}
          command:
          - /hyperkube
          - scheduler
          - --kubeconfig=/etc/kubernetes/kubeconfig/kube-scheduler.yaml
          - --authentication-kubeconfig=/etc/kubernetes/kubeconfig/kube-controller-manager.yaml
          - --authorization-kubeconfig=/etc/kubernetes/kubeconfig/kube-controller-manager.yaml

and

  - path: /etc/kubernetes/manifests/kube-scheduler.yaml
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-scheduler
        namespace: kube-system
        labels:
          k8s-app: kube-scheduler
      spec:
        hostNetwork: true
        containers:
        - name: kube-scheduler
          image: {{.HyperkubeImage.RepoWithTag}}
          command:
          - /hyperkube
          - scheduler
          - --kubeconfig=/etc/kubernetes/kubeconfig/kube-scheduler.yaml
          - --authentication-kubeconfig=/etc/kubernetes/kubeconfig/kube-scheduler.yaml
          - --authorization-kubeconfig=/etc/kubernetes/kubeconfig/kube-scheduler.yaml

it seems both kube-scheduler and kube-controller are in system namespace, but it is required to access kube-system namespace according to the error msg?
I0409 04:53:17.081295       1 serving.go:318] Generated self-signed cert in-memory
W0409 04:53:24.285379       1 authentication.go:262] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0409 04:53:24.285397       1 authentication.go:199] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-controller-manager" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0409 04:53:24.285407       1 authentication.go:200] Continuing without authentication configuration. This may treat all requests as anonymous.
W0

@daili Thanks for your effort!
Sorry I can't provide a direct anwser, but I now suspect you need to pass --use-service-account-credentials to controller-manager according to https://kubernetes.io/docs/reference/access-authn-authz/rbac/#controller-roles
Otherwise you need to create appropriate clusterrolebindings to give kube-controller-manager user(specified in the controller-manager's spec) enough permissions, as described in https://kubernetes.io/docs/reference/access-authn-authz/rbac/#core-component-roles
If your apiserver is already up and running, could you try running kubectl -n kube-system get clusterrole -o yaml and kubeclt -n kube-system clusterrolebinding and anything you might find useful, to dump what your cluster already have?

Hi, just to add my opinion on this - I think that we are not ready for thinking about rolling out 1.14 via kube-aws just yet.  It's great that you have identified this issue!  I would like to concentrate on releasing our 0.13 release with the latest k8s 1.13 release and work through these security changes, because they will have a knock on effect on a number of services that we have created which presently rely on the unauthenticated api server endpoint.

@davidmccormick Thanks for your response. Looking forward to the new release!

Just my two cents, we've 0.12.3 running kubernetesVersion: v1.14.1 since a month ago. No issues so far, all looking good.

Hi, please can you check out our beta release https://github.com/kubernetes-incubator/kube-aws/releases/tag/v0.14.0-rc.1 which upgrades kubernetes to 1.14.2 but also includes a large number of features and enhancements.  Whilst it may be possible to run a 1.14 cluster using the 0.12.3 release I would not recommend it - and you'll be missing out on some of the work we've done towards better security, elastic-spot fleets, PodPriority and TLSBootstrapping on the controllers.

@davidmccormick I saw that we're up to v0.14.0-rc.3 released 7/12/19.  Do you have an idea how close we are to a GA release of the v0.14.x trunk (or v0.13.x)?  I would love to be able to get K8s 1.14 into prod.
/remove-lifecycle stale

Very close now - v0.13.0 and v0.14.0 will be the next releases

Closing this as v0.13.0 and v0.14.0 are now out -please re-open if you continue to see an issue.