panic / v1alpha2._RuntimeService / SIGSEGV code=0x1 addr=0x38 pc=0x55f266891275

bug report
Description
on startup, after 'start streaming server' and a few operations, containerd crashloops almost exactly every minute at same seconds and prints the following stacktrace
May 29 12:19:01 minionpublic1 containerd[9691]: time="2019-05-29T12:19:01.096721955Z" level=warning msg="The image sha256:f0fad859c909baef1b038ef8d2f6e76fc252e25a3d9af37
May 29 12:19:01 minionpublic1 containerd[9691]: time="2019-05-29T12:19:01.119703566Z" level=info msg="Start event monitor"
May 29 12:19:01 minionpublic1 containerd[9691]: time="2019-05-29T12:19:01.119822706Z" level=info msg="Start snapshots syncer"
May 29 12:19:01 minionpublic1 containerd[9691]: time="2019-05-29T12:19:01.119834583Z" level=info msg="Start streaming server"
May 29 12:19:10 minionpublic1 containerd[9691]: time="2019-05-29T12:19:08.807256438Z" level=info msg="RemoveContainer for "1cde6af58b9e9ec2ea4481d7ece8314a701bb0a67ba649
May 29 12:19:10 minionpublic1 containerd[9691]: time="2019-05-29T12:19:08.807491426Z" level=info msg="RemoveContainer for "1cde6af58b9e9ec2ea4481d7ece8314a701bb0a67ba649
May 29 12:19:10 minionpublic1 containerd[9691]: panic: runtime error: invalid memory address or nil pointer dereference
May 29 12:19:10 minionpublic1 containerd[9691]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x55fe3e32f275]
May 29 12:19:10 minionpublic1 containerd[9691]: goroutine 85 [running]:
May 29 12:19:10 minionpublic1 containerd[9691]: github.com/containerd/containerd.WithSnapshotCleanup(0x55fe3f2ab0c0, 0xc00072c3c0, 0xc0003fc0b0, 0xc000208ac0, 0x40, 0xc0
May 29 12:19:10 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/container_opts.go:144 +0xd5
May 29 12:19:10 minionpublic1 containerd[9691]: github.com/containerd/containerd.(*container).Delete(0xc0006bcc80, 0x55fe3f2ab0c0, 0xc00072c3c0, 0xc00000e080, 0x1, 0x1,
May 29 12:19:10 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/container.go:152 +0x1a1
May 29 12:19:10 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).RemoveContainer(0xc000462480,
May 29 12:19:10 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_re
May 29 12:19:10 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*instrumentedService).RemoveContainer(0xc00
May 29 12:19:10 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/instrumented
May 29 12:19:10 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2._RuntimeService_RemoveCon
May 29 12:19:10 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1
May 29 12:19:08 minionpublic1 systemd[1]: containerd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 29 12:19:11 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x55fe3f2ab0c
May 29 12:19:11 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/serv
May 29 12:19:11 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2._RuntimeService_RemoveCon
May 29 12:19:11 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1
May 29 12:19:11 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000134000, 0x55fe3f2b5440, 0x
May 29 12:19:11 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1011 +0x4cf
May 29 12:19:11 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000134000, 0x55fe3f2b5440, 0xc00
May 29 12:19:11 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1249 +0x1313
May 29 12:19:11 minionpublic1 containerd[9691]: github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004d45a0, 0xc000134000,
May 29 12:19:11 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:680 +0xa1
May 29 12:19:11 minionpublic1 containerd[9691]: created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
May 29 12:19:11 minionpublic1 containerd[9691]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:678 +0xa3
May 29 12:19:08 minionpublic1 systemd[1]: containerd.service: Failed with result 'exit-code'.

Important to notice that when the k8s node comes up, it first starts with containerd 1.2.5 which is the one installed by Docker, lives for about 90 seconds without doing anything (as shown in logs, after successfully starting), then restarts at containerd 1.2.6 which is installed afterwards, then the panic-loop occurs.
After a while of crashlooping, an hour or so, it stops doing so.
Steps to reproduce the issue:
It only happens in one node out of many but the node changes, cannot pinpoint any configuration difference or specific node problem.
The most relevance is with v1alpha2._RuntimeService which I don't know what it exactly is. In slightly different configurations pods fail to start printing a similar error to v1alpha2 runtimeservice etc.
In the meantime between crashloops, it prints a lot of these messages:

May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 4427 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 4512 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 4560 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 4660 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 4810 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 4912 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 4954 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 5123 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 5246 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 5403 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 5531 (containerd-shim) in control group while starting unit. Ignoring.
May 29 12:44:40 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 12:44:40 minionpublic1 systemd[1]: containerd.service: Found left-over process 5634 (containerd-shim) in control group while starting unit. Ignoring.

and for other processes such as pods or s6-supervisor and others.
versions
containerd github.com/containerd/containerd v1.2.6 894b81a4b802e4eb2a91d1ce216b8817763c29fb

runc v1.0.0-rc8

k8s 1.11.8

config
io.containerd.runc.v1 shim
overlayfs
CNI 0.7.5

It panics at:
https://github.com/containerd/containerd/blob/v1.2.6/container_opts.go#L144
// WithSnapshotCleanup deletes the rootfs snapshot allocated for the container
func WithSnapshotCleanup(ctx context.Context, client *Client, c containers.Container) error {
	if c.SnapshotKey != "" {
		if c.Snapshotter == "" {
			return errors.Wrapf(errdefs.ErrInvalidArgument, "container.Snapshotter must be set to cleanup rootfs snapshot")
		}
		return client.SnapshotService(c.Snapshotter).Remove(ctx, c.SnapshotKey) // <---- This line
	}
	return nil
}

I think it is most likely that the snapshotter is not available. Can you provide the full containerd log? Maybe the overlay snapshotter is not successfully initialized.

Should containerd panic in this case? Or should we try to avoid panic?

Yes, first run is with docker+containerd and aufs due to a race in bootstrap, possibly leaves some unwanted artifacts and WithSnapshotCleanup is not able to clean them up. This probably won't be happening after docker is completely removed for new nodes.
Should it be able to clean up better, have more pathways to do so? Otherwise panic and restart looks fine if it can't do it.
overlayfs seems to be loading normally in logs.
Some questions on shims - is runc.v1 better than linux.v1? will runc.v2 be released with 1.3 with more bugfixes?
There is some default config on [plugins.linux] like no_shim etc, is there something similar for runc.v1?
How should I get more visibility in snapshotters and check intrinsics of aufs and overlayfs?
logs (quite verbose)

May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.184415728Z" level=info msg="starting containerd" revision=894b81a4b802e4eb2a91d1ce216b8817763c29fb version=v1.2.6
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.184781334Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.184816205Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.184846710Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.184949911Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185085106Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185104331Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185142170Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185160021Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185175786Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185188965Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185200686Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185212013Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185228603Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185263814Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.185372873Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.195066493Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.195805424Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.195889334Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196002194Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196059793Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196074464Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196086200Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196097970Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196110207Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196123013Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196134701Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196145475Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196224220Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196244656Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196259604Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196271417Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196569988Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runc.v1 Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress: StreamServerPort:10010 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384} ContainerdRootDir:/run/containerd/root ContainerdEndpoint:/run/containerd/state/containerd.sock RootDir:/run/containerd/root/io.containerd.grpc.v1.cri StateDir:/run/containerd/state/io.containerd.grpc.v1.cri}"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196597687Z" level=info msg="Connect containerd service"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.196788721Z" level=info msg="Get image filesystem path "/run/containerd/root/io.containerd.snapshotter.v1.overlayfs""
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.197789257Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.197840676Z" level=info msg="Start subscribing containerd event"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.197900147Z" level=info msg="Start recovering state"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.198077045Z" level=info msg=serving... address="/run/containerd/state/debug.sock"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.198127089Z" level=info msg=serving... address="<IP_ADDRESS>:1234"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.198179494Z" level=info msg=serving... address="/run/containerd/state/containerd.sock"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.198196345Z" level=info msg="containerd successfully booted in 0.014977s"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.826298075Z" level=info msg="Start event monitor"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.826390336Z" level=info msg="Start snapshots syncer"
May 29 17:26:44 minionpublic1 containerd[18290]: time="2019-05-29T17:26:44.826403132Z" level=info msg="Start streaming server"
May 29 17:26:48 minionpublic1 containerd[18290]: time="2019-05-29T17:26:48.115379246Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:26:48 minionpublic1 containerd[18290]: time="2019-05-29T17:26:48.258595472Z" level=info msg="Finish piping "stdout" of container exec "e340e328e3cae98c15ea6b09be249496a56946552fde58bc7ef5605ac414fe1d""
May 29 17:26:48 minionpublic1 containerd[18290]: time="2019-05-29T17:26:48.258599489Z" level=info msg="Finish piping "stderr" of container exec "e340e328e3cae98c15ea6b09be249496a56946552fde58bc7ef5605ac414fe1d""
May 29 17:26:48 minionpublic1 containerd[18290]: time="2019-05-29T17:26:48.259299409Z" level=info msg="Exec process "e340e328e3cae98c15ea6b09be249496a56946552fde58bc7ef5605ac414fe1d" exits with exit code 0 and error <nil>"
May 29 17:26:48 minionpublic1 containerd[18290]: time="2019-05-29T17:26:48.282711061Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:26:48 minionpublic1 containerd[18290]: time="2019-05-29T17:26:48.305545257Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:e340e328e3cae98c15ea6b09be249496a56946552fde58bc7ef5605ac414fe1d,Pid:18532,ExitStatus:0,ExitedAt:2019-05-29 17:26:48.259016474 +0000 UTC,}"
May 29 17:26:53 minionpublic1 containerd[18290]: time="2019-05-29T17:26:53.115456627Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:26:53 minionpublic1 containerd[18290]: time="2019-05-29T17:26:53.261418901Z" level=info msg="Finish piping "stderr" of container exec "1a36454120b1a8bbec6fa7ad1d5ce6837b082729d9403ad9a4322264313f48ff""
May 29 17:26:53 minionpublic1 containerd[18290]: time="2019-05-29T17:26:53.261432944Z" level=info msg="Finish piping "stdout" of container exec "1a36454120b1a8bbec6fa7ad1d5ce6837b082729d9403ad9a4322264313f48ff""
May 29 17:26:53 minionpublic1 containerd[18290]: time="2019-05-29T17:26:53.262069652Z" level=info msg="Exec process "1a36454120b1a8bbec6fa7ad1d5ce6837b082729d9403ad9a4322264313f48ff" exits with exit code 0 and error <nil>"
May 29 17:26:53 minionpublic1 containerd[18290]: time="2019-05-29T17:26:53.284020991Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:26:53 minionpublic1 containerd[18290]: time="2019-05-29T17:26:53.307153777Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:1a36454120b1a8bbec6fa7ad1d5ce6837b082729d9403ad9a4322264313f48ff,Pid:18611,ExitStatus:0,ExitedAt:2019-05-29 17:26:53.261800734 +0000 UTC,}"
May 29 17:26:58 minionpublic1 containerd[18290]: time="2019-05-29T17:26:58.115270747Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:26:58 minionpublic1 containerd[18290]: time="2019-05-29T17:26:58.253254886Z" level=info msg="Finish piping "stdout" of container exec "014e20d00abf6cc23a04d875e9e38579edb5f118422d5e0a12e55d2ffca214f3""
May 29 17:26:58 minionpublic1 containerd[18290]: time="2019-05-29T17:26:58.253351280Z" level=info msg="Finish piping "stderr" of container exec "014e20d00abf6cc23a04d875e9e38579edb5f118422d5e0a12e55d2ffca214f3""
May 29 17:26:58 minionpublic1 containerd[18290]: time="2019-05-29T17:26:58.253988938Z" level=info msg="Exec process "014e20d00abf6cc23a04d875e9e38579edb5f118422d5e0a12e55d2ffca214f3" exits with exit code 0 and error <nil>"
May 29 17:26:58 minionpublic1 containerd[18290]: time="2019-05-29T17:26:58.275783350Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:26:58 minionpublic1 containerd[18290]: time="2019-05-29T17:26:58.299724351Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:014e20d00abf6cc23a04d875e9e38579edb5f118422d5e0a12e55d2ffca214f3,Pid:18752,ExitStatus:0,ExitedAt:2019-05-29 17:26:58.253741034 +0000 UTC,}"
May 29 17:27:03 minionpublic1 containerd[18290]: time="2019-05-29T17:27:03.115355486Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:03 minionpublic1 containerd[18290]: time="2019-05-29T17:27:03.262451892Z" level=info msg="Finish piping "stdout" of container exec "5dd775417bcf9e08af154a47cc43dd0de87f3fef4b0b03037d882ddfbcb3a579""
May 29 17:27:03 minionpublic1 containerd[18290]: time="2019-05-29T17:27:03.262490368Z" level=info msg="Finish piping "stderr" of container exec "5dd775417bcf9e08af154a47cc43dd0de87f3fef4b0b03037d882ddfbcb3a579""
May 29 17:27:03 minionpublic1 containerd[18290]: time="2019-05-29T17:27:03.263058309Z" level=info msg="Exec process "5dd775417bcf9e08af154a47cc43dd0de87f3fef4b0b03037d882ddfbcb3a579" exits with exit code 0 and error <nil>"
May 29 17:27:03 minionpublic1 containerd[18290]: time="2019-05-29T17:27:03.284152520Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:03 minionpublic1 containerd[18290]: time="2019-05-29T17:27:03.311636811Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:5dd775417bcf9e08af154a47cc43dd0de87f3fef4b0b03037d882ddfbcb3a579,Pid:18865,ExitStatus:0,ExitedAt:2019-05-29 17:27:03.262847364 +0000 UTC,}"
May 29 17:27:08 minionpublic1 containerd[18290]: time="2019-05-29T17:27:08.115397192Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:08 minionpublic1 containerd[18290]: time="2019-05-29T17:27:08.266343144Z" level=info msg="Finish piping "stderr" of container exec "33cedfec7992657720b621f8ba3f34a74eb9c44e48a36154cf68e286d349f6d3""
May 29 17:27:08 minionpublic1 containerd[18290]: time="2019-05-29T17:27:08.266370485Z" level=info msg="Finish piping "stdout" of container exec "33cedfec7992657720b621f8ba3f34a74eb9c44e48a36154cf68e286d349f6d3""
May 29 17:27:08 minionpublic1 containerd[18290]: time="2019-05-29T17:27:08.267035056Z" level=info msg="Exec process "33cedfec7992657720b621f8ba3f34a74eb9c44e48a36154cf68e286d349f6d3" exits with exit code 0 and error <nil>"
May 29 17:27:08 minionpublic1 containerd[18290]: time="2019-05-29T17:27:08.288841587Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:08 minionpublic1 containerd[18290]: time="2019-05-29T17:27:08.312142700Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:33cedfec7992657720b621f8ba3f34a74eb9c44e48a36154cf68e286d349f6d3,Pid:18941,ExitStatus:0,ExitedAt:2019-05-29 17:27:08.266702541 +0000 UTC,}"
May 29 17:27:13 minionpublic1 containerd[18290]: time="2019-05-29T17:27:13.115301929Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:13 minionpublic1 containerd[18290]: time="2019-05-29T17:27:13.261747431Z" level=info msg="Finish piping "stderr" of container exec "e382c46f6712160d512b64a67e36c4eea951a5daaf57a21aab0f4c5e4af902dd""
May 29 17:27:13 minionpublic1 containerd[18290]: time="2019-05-29T17:27:13.261825325Z" level=info msg="Finish piping "stdout" of container exec "e382c46f6712160d512b64a67e36c4eea951a5daaf57a21aab0f4c5e4af902dd""
May 29 17:27:13 minionpublic1 containerd[18290]: time="2019-05-29T17:27:13.264489575Z" level=info msg="Exec process "e382c46f6712160d512b64a67e36c4eea951a5daaf57a21aab0f4c5e4af902dd" exits with exit code 0 and error <nil>"
May 29 17:27:13 minionpublic1 containerd[18290]: time="2019-05-29T17:27:13.286036457Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:13 minionpublic1 containerd[18290]: time="2019-05-29T17:27:13.310321697Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:e382c46f6712160d512b64a67e36c4eea951a5daaf57a21aab0f4c5e4af902dd,Pid:19098,ExitStatus:0,ExitedAt:2019-05-29 17:27:13.2626286 +0000 UTC,}"
May 29 17:27:18 minionpublic1 containerd[18290]: time="2019-05-29T17:27:18.115386742Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:18 minionpublic1 containerd[18290]: time="2019-05-29T17:27:18.251116165Z" level=info msg="Finish piping "stderr" of container exec "14ff2d7d9a58eba6ce3b3162d7a4db4f986ba82ffa98e60e6cb273b7f0138131""
May 29 17:27:18 minionpublic1 containerd[18290]: time="2019-05-29T17:27:18.251167176Z" level=info msg="Finish piping "stdout" of container exec "14ff2d7d9a58eba6ce3b3162d7a4db4f986ba82ffa98e60e6cb273b7f0138131""
May 29 17:27:18 minionpublic1 containerd[18290]: time="2019-05-29T17:27:18.252010365Z" level=info msg="Exec process "14ff2d7d9a58eba6ce3b3162d7a4db4f986ba82ffa98e60e6cb273b7f0138131" exits with exit code 0 and error <nil>"
May 29 17:27:18 minionpublic1 containerd[18290]: time="2019-05-29T17:27:18.274078900Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:18 minionpublic1 containerd[18290]: time="2019-05-29T17:27:18.297812407Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:14ff2d7d9a58eba6ce3b3162d7a4db4f986ba82ffa98e60e6cb273b7f0138131,Pid:19217,ExitStatus:0,ExitedAt:2019-05-29 17:27:18.251700409 +0000 UTC,}"
May 29 17:27:23 minionpublic1 containerd[18290]: time="2019-05-29T17:27:23.115331950Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:23 minionpublic1 containerd[18290]: time="2019-05-29T17:27:23.279410639Z" level=info msg="Finish piping "stderr" of container exec "626015c38588a334908fbf63f7e8939e2846601219f32b5239348fc2835e72ba""
May 29 17:27:23 minionpublic1 containerd[18290]: time="2019-05-29T17:27:23.279485506Z" level=info msg="Finish piping "stdout" of container exec "626015c38588a334908fbf63f7e8939e2846601219f32b5239348fc2835e72ba""
May 29 17:27:23 minionpublic1 containerd[18290]: time="2019-05-29T17:27:23.280216013Z" level=info msg="Exec process "626015c38588a334908fbf63f7e8939e2846601219f32b5239348fc2835e72ba" exits with exit code 0 and error <nil>"
May 29 17:27:23 minionpublic1 containerd[18290]: time="2019-05-29T17:27:23.301218854Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:23 minionpublic1 containerd[18290]: time="2019-05-29T17:27:23.326541116Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:626015c38588a334908fbf63f7e8939e2846601219f32b5239348fc2835e72ba,Pid:19295,ExitStatus:0,ExitedAt:2019-05-29 17:27:23.279891774 +0000 UTC,}"
May 29 17:27:28 minionpublic1 containerd[18290]: time="2019-05-29T17:27:28.115300519Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:28 minionpublic1 containerd[18290]: time="2019-05-29T17:27:28.251032177Z" level=info msg="Finish piping "stderr" of container exec "d5bbf69ad227b1ee26802b5f892755385cf769b040f03d8fe77a2df24c6ec479""
May 29 17:27:28 minionpublic1 containerd[18290]: time="2019-05-29T17:27:28.251108774Z" level=info msg="Finish piping "stdout" of container exec "d5bbf69ad227b1ee26802b5f892755385cf769b040f03d8fe77a2df24c6ec479""
May 29 17:27:28 minionpublic1 containerd[18290]: time="2019-05-29T17:27:28.251825422Z" level=info msg="Exec process "d5bbf69ad227b1ee26802b5f892755385cf769b040f03d8fe77a2df24c6ec479" exits with exit code 0 and error <nil>"
May 29 17:27:28 minionpublic1 containerd[18290]: time="2019-05-29T17:27:28.274281955Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:28 minionpublic1 containerd[18290]: time="2019-05-29T17:27:28.296971638Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:d5bbf69ad227b1ee26802b5f892755385cf769b040f03d8fe77a2df24c6ec479,Pid:19439,ExitStatus:0,ExitedAt:2019-05-29 17:27:28.251490065 +0000 UTC,}"
May 29 17:27:33 minionpublic1 containerd[18290]: time="2019-05-29T17:27:33.115342662Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:33 minionpublic1 containerd[18290]: time="2019-05-29T17:27:33.252100262Z" level=info msg="Finish piping "stdout" of container exec "77901e79be10f2ec1dd6c3158c8bfb45eaa0646eb2b5d0a5bc7857b81a795250""
May 29 17:27:33 minionpublic1 containerd[18290]: time="2019-05-29T17:27:33.252101269Z" level=info msg="Finish piping "stderr" of container exec "77901e79be10f2ec1dd6c3158c8bfb45eaa0646eb2b5d0a5bc7857b81a795250""
May 29 17:27:33 minionpublic1 containerd[18290]: time="2019-05-29T17:27:33.254771579Z" level=info msg="Exec process "77901e79be10f2ec1dd6c3158c8bfb45eaa0646eb2b5d0a5bc7857b81a795250" exits with exit code 0 and error <nil>"
May 29 17:27:33 minionpublic1 containerd[18290]: time="2019-05-29T17:27:33.276446249Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:33 minionpublic1 containerd[18290]: time="2019-05-29T17:27:33.295221090Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:77901e79be10f2ec1dd6c3158c8bfb45eaa0646eb2b5d0a5bc7857b81a795250,Pid:19550,ExitStatus:0,ExitedAt:2019-05-29 17:27:33.252871527 +0000 UTC,}"
May 29 17:27:38 minionpublic1 containerd[18290]: time="2019-05-29T17:27:38.115203031Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:38 minionpublic1 containerd[18290]: time="2019-05-29T17:27:38.263787837Z" level=info msg="Finish piping "stderr" of container exec "5656da6224dc223af70ea53c1ce46aac1141b1ebc0d1c75628c838daeb7ac56a""
May 29 17:27:38 minionpublic1 containerd[18290]: time="2019-05-29T17:27:38.263791061Z" level=info msg="Finish piping "stdout" of container exec "5656da6224dc223af70ea53c1ce46aac1141b1ebc0d1c75628c838daeb7ac56a""
May 29 17:27:38 minionpublic1 containerd[18290]: time="2019-05-29T17:27:38.264637959Z" level=info msg="Exec process "5656da6224dc223af70ea53c1ce46aac1141b1ebc0d1c75628c838daeb7ac56a" exits with exit code 0 and error <nil>"
May 29 17:27:38 minionpublic1 containerd[18290]: time="2019-05-29T17:27:38.287420436Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" returns with exit code 0"
May 29 17:27:38 minionpublic1 containerd[18290]: time="2019-05-29T17:27:38.308004623Z" level=info msg="TaskExit event &TaskExit{ContainerID:efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513,ID:5656da6224dc223af70ea53c1ce46aac1141b1ebc0d1c75628c838daeb7ac56a,Pid:19608,ExitStatus:0,ExitedAt:2019-05-29 17:27:38.264348275 +0000 UTC,}"
May 29 17:27:43 minionpublic1 containerd[18290]: time="2019-05-29T17:27:43.115322552Z" level=info msg="ExecSync for "efbfe8b1f96aa40dca72e9b4e92e77a80441fc3503f10d687cdcd471e005c513" with command [./probe.sh] and timeout 5 (s)"
May 29 17:27:43 minionpublic1 containerd[18290]: time="2019-05-29T17:27:43.139811734Z" level=info msg="RemoveContainer for "102ce537bacbdd5efa29b10ad6a95e584598f2585e1ee73a2e0aa745a769cf75""
May 29 17:27:43 minionpublic1 containerd[18290]: time="2019-05-29T17:27:43.140054684Z" level=info msg="RemoveContainer for "102ce537bacbdd5efa29b10ad6a95e584598f2585e1ee73a2e0aa745a769cf75" returns successfully"
May 29 17:27:43 minionpublic1 containerd[18290]: panic: runtime error: invalid memory address or nil pointer dereference
May 29 17:27:43 minionpublic1 containerd[18290]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x5642f58b4275]
May 29 17:27:43 minionpublic1 containerd[18290]: goroutine 701 [running]:
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd.WithSnapshotCleanup(0x5642f68300c0, 0xc000983cb0, 0xc00011ba20, 0xc000714800, 0x40, 0xc000983f80, 0xc0007abc80, 0x3d, 0xc00094c240, 0x1e, ...)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/container_opts.go:144 +0xd5
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd.(*container).Delete(0xc0007b26c0, 0x5642f68300c0, 0xc000983cb0, 0xc00000e5f8, 0x1, 0x1, 0xc0003e16c0, 0xc000040e60)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/container.go:152 +0x1a1
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).RemoveContainer(0xc00012a900, 0x5642f68300c0, 0xc000983cb0, 0xc0006fd270, 0x0, 0x0, 0x0)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_remove.go:67 +0x4b0
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*instrumentedService).RemoveContainer(0xc00034c290, 0x5642f68300c0, 0xc000983b90, 0xc0006fd270, 0x0, 0x0, 0x0)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/instrumented_service.go:232 +0x16c
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2._RuntimeService_RemoveContainer_Handler.func1(0x5642f68300c0, 0xc000983b90, 0x5642f675a240, 0xc0006fd270, 0xc0006254a0, 0x5642f666e840, 0x5642f72e3210, 0xc000bf7aa0)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2/api.pb.go:4316 +0x88
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x5642f68300c0, 0xc000983b90, 0x5642f675a240, 0xc0006fd270, 0xc000584860, 0xc000584880, 0x5642f666e840, 0x5642f72e3210, 0x5642f67c7800, 0xc0007a0a50)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xa6
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2._RuntimeService_RemoveContainer_Handler(0x5642f67ddaa0, 0xc00034c290, 0x5642f68300c0, 0xc000983b90, 0xc00022e690, 0x5642f680ad60, 0x0, 0x0, 0x2c2d69c3, 0x7)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2/api.pb.go:4318 +0x15a
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000cc000, 0x5642f683a440, 0xc000230000, 0xc0007a0a50, 0xc0003235c0, 0x5642f7276498, 0x0, 0x0, 0x0)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1011 +0x4cf
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0000cc000, 0x5642f683a440, 0xc000230000, 0xc0007a0a50, 0x0)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1249 +0x1313
May 29 17:27:43 minionpublic1 containerd[18290]: github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00010e780, 0xc0000cc000, 0x5642f683a440, 0xc000230000, 0xc0007a0a50)
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:680 +0xa1
May 29 17:27:43 minionpublic1 containerd[18290]: created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
May 29 17:27:43 minionpublic1 containerd[18290]:         /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:678 +0xa3
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Failed with result 'exit-code'.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Service hold-off time over, scheduling restart.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Scheduled restart job, restart counter is at 287.
May 29 17:27:43 minionpublic1 systemd[1]: Stopped containerd container runtime.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4427 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4512 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4560 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4660 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4810 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4912 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4954 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5123 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5246 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5403 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5531 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5634 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6395 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4458 (pause) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4539 (pause) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4687 (pause) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4597 (kube-proxy) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4851 (pause) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 4989 (flanneld) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5019 (pause) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5155 (sh) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 17496 (sleep) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5273 (traefik) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5429 (pause) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5571 (tini) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5617 (fluentd) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6178 (ruby2.3) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5664 (s6-svscan) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 5719 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6050 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6051 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6052 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6053 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6054 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6055 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6056 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6057 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6062 (s6-format-filte) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6063 (agent) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6065 (s6-format-filte) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6066 (process-agent) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6067 (s6-format-filte) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6068 (trace-agent) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6423 (sh) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: containerd.service: Found left-over process 6452 (node-problem-de) in control group while starting unit. Ignoring.
May 29 17:27:43 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:43 minionpublic1 systemd[1]: Starting containerd container runtime...
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4427 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4512 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4560 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4660 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4810 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4912 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4954 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5123 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5246 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5403 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5531 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5634 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6395 (containerd-shim) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4458 (pause) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4539 (pause) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4687 (pause) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4597 (kube-proxy) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4851 (pause) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 4989 (flanneld) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5019 (pause) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5155 (sh) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 17496 (sleep) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5273 (traefik) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5429 (pause) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5571 (tini) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5617 (fluentd) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6178 (ruby2.3) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5664 (s6-svscan) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 5719 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6050 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6051 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6052 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6053 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6054 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6055 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6056 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6057 (s6-supervise) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6062 (s6-format-filte) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6063 (agent) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6065 (s6-format-filte) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6066 (process-agent) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6067 (s6-format-filte) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6068 (trace-agent) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6423 (sh) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: containerd.service: Found left-over process 6452 (node-problem-de) in control group while starting unit. Ignoring.
May 29 17:27:44 minionpublic1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 29 17:27:44 minionpublic1 systemd[1]: Started containerd container runtime.
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.200615967Z" level=info msg="starting containerd" revision=894b81a4b802e4eb2a91d1ce216b8817763c29fb version=v1.2.6
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201015518Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201046711Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201080480Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201159617Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201323949Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201346825Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201418643Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201433335Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201447790Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201471459Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201489504Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201508760Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201525673Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201547217Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
May 29 17:27:44 minionpublic1 containerd[19762]: time="2019-05-29T17:27:44.201667707Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2

Some questions on shims - is runc.v1 better than linux.v1? will runc.v2 be released with 1.3 with more bugfixes?

runc.v1 is similar with linux.v1, but newer. It is an implementation of the new shim v2 API https://github.com/containerd/containerd/issues/2426.
runc.v2 is better for Kubernetes integration, because it only starts one shim per pod.

There is some default config on [plugins.linux] like no_shim etc, is there something similar for runc.v1?

No. Shim is required for shim v2 for now I think.

How should I get more visibility in snapshotters and check intrinsics of aufs and overlayfs?

ctr snapshots?


Zenburn selection background low contrast

Theme(s)
doom-zenburn
What were you expecting?
The selected item background contrast is too low. This is similar to issues:
https://github.com/doomemacs/themes/issues/652
https://github.com/doomemacs/themes/issues/669
What actually happened?
Here is what you see in find file (or execute command) with doom-zenburn:

Here it is with zenburn (note that at least there is an underline):

Here it is with no theme:

The color that it displays when you mouse-over also varies for each theme.
Here is what you see for completion with doom-zenburn:

Completion with zenburn:

Completion with no theme:

I can barely make out the highlight region at all (it's over the second option). I'm guessing the colors are identical as with find-file, but it is probably more visible in find file due to the surrounding colors. Also, notice that there is no border with completions and the background color is identical, so completions looks like it's part of the main buffer. In a Java file, the completion region is a different shade.
Finally, maybe a different issue, but when you reload the theme, some faces get messed up. Reload doom-zenburn and then find-file:

The border is gone, and it again looks like find file is in the elisp buffer. While trying to find a theme, I noticed the column numbers ended up a different color from the rest of the buffer and the fringe I set with posframe ended up a different color. I don't know how to reproduce. It's probably due to something not being properly reset.
I've seen  some screenshots and videos where the theme was clearly a dark mode, but the highlight color was bright, or at least not a shade of grey. Most of the dark modes I've tried seem to have this issue with the selection being almost the same as the background. It may be due to Ivy; I don't know at this point.
I've  switched to dark+ and customized my foreground and background colors. It's a little better. The selection color background is a blueish color. There's still no way to distinguish the completions from the rest of the buffer.
Steps to reproduce
Install Ivy and Doom Themes.
Switch to doom-zenburn.
Open a file or invoke completion in elisp mode.
I'm not using the latest commit, but if necessary I could test again. I'm also not sure if Ivy is relevant or not. I also modified the minibuffer with posframe,  but this doesn't change the highlight color. I can provide the code if necessary.
Operating system
Windows 10
Emacs version
Emacs 28.1
Installed commit of doom-themes
Melpa Stable 2.3.0

I forgot about the ui-control in dap. This is with dark+, but it's similar with zenburn.
Doom-dark+:

No theme:

Load doom-dark+ after loading ui-controls.:

Here's doom-zenburn tabs:

No theme tabs:

Also, I noticed in doom-dark+, I get the an error about invalid face :background nil due to centaur tabs. See this discussion. Specifically, it's centaur-tabs-unselected-modified that has unspecified background. The problem doesn't exist with doom-zenburn.

Here is what the completion looks like in doom-dark+/corfu with my slightly modified background color. Having a thin border is better than no visual clue there is a child window, but images I've seen with a slightly different shade child window look better in my opinion.

The ediff faces are too muted for me in doom-dark+. I commented out all the ediff lines in doom-themes-base.el. The default ediff faces are too intense. But if I have to choose between the two, I would choose too intense because at least I can see them! Is there any way to disable overwriting the default faces?


Revert commits related to _azure_BUILD_FOR_TESTING and _azure_*_FOR_TESTS

I encourage @ahsonkhan to re-apply some parts of it as soon as possible.
In my opinion, these were the good things:

Renaming TESTING_BUILD to _azure_TESTING_BUILD. We can solve this by adding if(BUILD_TESTING) add_compile_definitions(_azure_TESTING_BUILD) in a centralized location, such as AzureGlobalCompileOptions.cmake/AddGoogleTest.cmake (or make a new cmake file). Also, the existing #if defined/ #ifdef(_azure_TESTING_BUILD_AMQP) can be switched to #if defined(_azure_TESTING_BUILD)  after that.
All libraries having _azure_TESTING_BUILD defined when building the tests (#5414). If Item 1 above is done in a centralized
way, i.e. via .cmake file, this item should be automatically taken care of as well.

For later:
4. Maybe drop "ifndef TESTING final. // BTW, for test_hooks.hpp, should you end up creating one, I think, testing.hppwould be even better name, because the name implies we may put more testing-related stuff there in the future, the name is more generic. 5. Maybe find a way to still dropvirtualandfinal` from Doxygen; maybe from ApiView.
6. Maybe think of the mocking support / exposing internal details or test hooks in another way / avoiding (so many) friend declarations (#5147), and applying that pattern universally.

/azp run cpp - core

/azp run cpp - attestation

/azp run cpp - storage

/azp run cpp - identity

/azp run cpp - keyvault

/azp run cpp - tables

@ahsonkhan, @LarryOsterman  - I enumerated all things in this PR description, please comment if you think we should capture more.
I deliberately not adding more original commits here, the PR is simple automated revert of 4 commits.


I dont have a password connected to my account (Sweden)🐛

What version of Integration has the issue?
V2.8.3
System Health details
System Information

version
core-2023.8.3

installation_type
Home Assistant Container

dev
false

hassio
false

docker
true

user
root

virtualenv
false

python_version
3.11.4

os_name
Linux

os_version
5.10.0-23-amd64

arch
x86_64

timezone
Europe/Stockholm

config_dir
/config

Home Assistant Community Store

GitHub API
ok

GitHub Content
ok

GitHub Web
ok

GitHub API Calls Remaining
5000

Installed Version
1.32.1

Stage
running

Available Repositories
1272

Downloaded Repositories
34

Home Assistant Cloud

logged_in
true

subscription_expiration
September 17, 2023 at 2:00 AM

relayer_connected
true

relayer_region
eu-central-1

remote_enabled
true

remote_connected
true

alexa_enabled
false

google_enabled
true

remote_server
eu-central-1-9.ui.nabu.casa

certificate_status
ready

can_reach_cert_server
ok

can_reach_cloud_auth
ok

can_reach_cloud
ok

Dashboards

dashboards
2

resources
13

views
27

mode
storage

Recorder

oldest_recorder_run
August 9, 2023 at 10:37 AM

current_recorder_run
August 22, 2023 at 1:47 PM

estimated_db_size
5080.94 MiB

database_engine
mysql

database_version
10.11.2

Spotify

api_endpoint_reachable
ok

Checklist

[X] I have enabled debug logging for my installation.
[X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).

Describe the issue
Just bought a watch for my daugher.
Via xplora.se enabled the watch/eSIM and got the app working.
When logging into xplora.se "mina sidor" (My pages) I do not have a password, a login token is generated and send via SMS to the "parent registrered cellphone number" at every login.
Have I missed anything? Do I need to (also) join the watch to another portal with xplora? Its all a bit confusing.
This component looks awesome and I cannot wait to get it going! <3
Reproduction steps

Buy a watch in Sweden
Join and enable it via xplora.se
Try installing the custom component and run the wizard - You will notice you dont have a password to enter. ;)
...

Debug logs
N/A

Regged a account on https://goplay.myxplora.com/ and the watch showed up there. :)
Not so streamlined user experiance from Xplora here... :)


schilytools

A better source of common utilities than toybox or busybox if you don't mind the CDDL. It's userland, not the kernel, so I don't think the license matters.
https://codeberg.org/schilytools/schilytools

Schilytools are cool, however it does not offer a complete userland.
Stuff like cp, mkdir, ls and other needed utils are not inclused in it

Make a PR in Frankensrc if you wanna! It’s easy to modify stuff if you want.


Typo

"ElementAspect instances are owned a single Element" should be "ElementAspect instances are owned by a single Element"

⚠ Do not edit this section. It is required for imodeljs.github.io ➟ GitHub issue linking

LINK: Fabric-of-the-Universe---iModel.js-link-to-docs
TAG: imodeljs-docs-question
SOURCE: https://imodeljs.github.io/iModelJs-docs-output/bis/intro/fabric-of-the-universe/

@ELymar Thanks for notifying us. I will inform the docs team.


Allow configuration of multiple networks via Nerves.Network.setup/2

I rebased @tmecklem 's change and updated it to work with the current version of Nerves Network.
I've tested two networks, with varying combinations of settings and it seems to work.
By nesting multiple wireless networks under the :network key, setup
will add and enable each network and then trigger WpaSupplicant to
scan and connect to the available network with the highest priority.
This maintains backwards compatibility with single wireless network
configuration.

Okay now that nerves_wpa_supplicant 0.4.0 is out this PR is ready to be merged.

TODO about converting WEP

I want to tackle this in a separate pull request along with the one on nerves_wpa_supplicant. It has the very real possibility of being an API breaking change and i'd like to address as much of these changes at once.


Remove duplicate util that checks for app only permissions

I noticed that we currently have 2 util functions that let us determine whether an access token has app only permissions or not.
1st util function:
https://github.com/pnp/cli-microsoft365/blob/fa2dfa85d34e501ff26a7d8525542e41b23fda07/src/Auth.ts#L682-L704
2nd util function:
https://github.com/pnp/cli-microsoft365/blob/fa2dfa85d34e501ff26a7d8525542e41b23fda07/src/utils/accessToken.ts#L2-L23

I suggest that we remove option 1 and update option 2 to return boolean | undefined.

I feel like this util function fits best in the accessToken.ts util (option 2).
What do you think @pnp/cli-for-microsoft-365-maintainers?

+1 on removing the duplicates and sticking with option 2.

Option #2 is simplified but it introduces a side-effect where we assume delegated auth when something goes wrong when reading a token. We should have a closer look if this won't lead to side-effects that in turn will cause exceptions or worse, undesired behavior.

Seems like 1 command is checking actually if the result is undefined. So I suggest that we remove option 1 and update option 2 to return boolean | undefined.

Could I work on this one?

Please do!


Integrating with paged.js

Hi!
Currently we are using paged.js (pagedjs-cli) to generate pdfs using https://www.pagedjs.org/ since it supports a lot of new css to control the output that chrome doesn't support natively.
However, it's pretty slow to start a new node/chrome instance each time so I'm looking for alternatives.
I tried to instead using the pagedjs polyfill js in the web page and the following option (since that node is generated by pagedjs)
wait_for: %{selector: ".pagedjs_pages", attribute: "style"}
However, it seems it does not get a reply and timeouts. I guess this is because the node with class=".pagedjs_pages" does not exist at the start.
The corresponding code in pagedjs-cli is here: https://gitlab.pagedmedia.org/tools/pagedjs-cli/blob/master/src/printer.js#L220
await page.waitForSelector(".pagedjs_pages");
Any help would be much appreciated!

Hello @norpan,
it seems that paged has an after hook that could be used for doing things after it has finished. Maybe you can create your own element in the DOM and add an attribute to the element with the after hook and use thewait_for option to wait for this element. Also note that the wait_for option is brand new and was just merged in today :), so maybe you have to use chromic_pdf from the main branch.
Cheers,
Andi

Hello @norpan,
it seems that paged has an after hook that could be used for doing things after it has finished. Maybe you can create your own element in the DOM and add an attribute to the element with the after hook and use thewait_for option to wait for this element. Also note that the wait_for option is brand new and was just merged in today :), so maybe you have to use chromic_pdf from the main branch.
Cheers,
Andi

Thanks, I hadn't considered that. I couldn't get the hook to work, but registering a handler and setting an attribute on the body worked. For anybody who has the same problem, here is what I did:
<script>
class PagedJsHandler extends Paged.Handler {
    constructor(chunker, polisher, caller) {
        super(chunker, polisher, caller);
    }

    afterRendered(pages) {
        document.body.setAttribute("rendered", "true")
    }
}
Paged.registerHandlers(PagedJsHandler)
</script>

and
          ChromicPDF.print_to_pdf({:url, "file://#{html_file}"},
            output: output_file,
            wait_for: %{selector: "body", attribute: "rendered"}
          )

Also, I had to increase the timeout for the session pool.

Thanks, I hadn't considered that. I couldn't get the hook to work, but registering a handler and setting an attribute on the body worked. For anybody who has the same problem, here is what I did:
<script>
class PagedJsHandler extends Paged.Handler {
    constructor(chunker, polisher, caller) {
        super(chunker, polisher, caller);
    }

    afterRendered(pages) {
        document.body.setAttribute("rendered", "true")
    }
}
Paged.registerHandlers(PagedJsHandler)
</script>

and
          ChromicPDF.print_to_pdf({:url, "file://#{html_file}"},
            output: output_file,
            wait_for: %{selector: "body", attribute: "rendered"}
          )

Also, I had to increase the timeout for the session pool.

The constructor isn't needed, is it?

The constructor isn't needed, is it?

I don't know, I just followed the example :)

I don't know, I just followed the example :)


Improve documentation for customizing text

Thanks for the great project.  Here is a suggestion to improve the documentation on how to customize the text:
text: "",                       // custom notification html text.  Optionally include up to two placeholders "%s"
                                  // which will be replaced with the browser version and contents of the link tag.
                                  // Example: "Your browser (%s) is old.  Please <a%s>update</a>"

Great idea. I will change this when I update other localization strings in the next months.


[BUGFIX] remove sent embedding

Description
I'm sorry. I was not carefully reading the documentation. pooled_out is not sent embedding. it is a tensor of pooled representation of the first tokens.
Checklist
Essentials

[x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage
[x] Code is well-documented

Changes

[x] remove sent embedding (pooled_output)

Job PR-618/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-618/1/index.html

looks like the CI was broken because tornado was silently upgraded from 5.1.1 to 6.0.1.

Rebased this branch.

@imgarylai I put the fix in #617. Once it's merged, we will need to rebase this PR.

@imgarylai I put the fix in #617. Once it's merged, we will need to rebase this PR.

no problem. I will rebase it again.

Job PR-618/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-618/4/index.html


Removed VSTS name in favor of Azure DevOps name

@exolvi Thank you for for fixing this. I am merging this PR and it should be live soon.


disable sass-lint-result.xml creation when there are no sass issues.

When there are no Sass issues, the empty sass-lint-result.xml file is being created. This makes Jenkins build failing - trying to parse an empty file. The file should not be generated in such case or it should be valid empty XML file. As requested by @hindsholm

I assume you mean scss-lint-result.xml, not sass-lint-result.xml :)
The gulp-scss-lint documentation says that you just need to add a property to the configuration in order to generate Checkstyle-formatted output:
gulp.src(['**/*.scss'])
  .pipe(scsslint({
    'reporterOutputFormat': 'Checkstyle',
  }))

This might also resolve this issue.

yes, of course scss-lint-result.xml :)

adding 'reporterOutputFormat': 'Checkstyle' will produce an empty XML:
<?xml version="1.0" encoding="utf-8"?>
<checkstyle version="1.5.6"></checkstyle>

but the mandatory gem needs to be installed:
Error: You need to have the scss_lint_reporter_checkstyle gem installed
updating readme with this and closing the issue on master. npm release soon.


How to import other .less files not from antd

Hi @SolidZORO , I have a problem I want to import other .less filess....
In development mode using this syntax, it works import './my-path/my-styles.less'
But when building the project these styles are not built and present in the build folder....also no errors appears.
Another interesting thing.....it works only when I'm using require('./my-path/my-styles.less').
So why it works with require but not with import?
And how to fix it??

The problem I remember from previous versions of next.js is that you must import the css file using the require syntax in _app.js or somewhere, else it won't work on prod. This may be a node server syntax issue, I'm not sure why at the moment, I'll deal with it later.


q.slice with two arguments does not become an array type

It returns a type with the fields from q.grab instead of an array type. It queries the data fine, but using two arguments gives the same type as just the one argument.

This is my query:
q(
  "*",
  q.filter("_type == 'news'"),
  q.grab({
    _id: q.string(),
    title: q.string(),
  }),
  q.slice(0, 2)
)

And this is the schema:
export default defineType({
  name: "news",
  type: "document",
  title: "News",
  fields: [
    defineField({
      name: "title",
      type: "string",
      title: "Title",
    }),
  ]
})


Add support for flipHorizontal animation

What does this PR do?

Adds support for flipHorizontal animation (which got lost with version 1.1.2 onward)

Looks good!
So swift 4 / xcode 9, no longer has this animation included?
Anyways, it's better to use it directly from Presentr as a custom PresentrAnimation.
Will test it out sometime today when I get the chance and if all is good will merge.
Thanks!!

Sorry. Been really busy with work.
Will get a couple of weeks of for the holidays and will catch up on issues and pull requests. Thanks for your patience.


JDBC wrapper Increasing CPU utilization and API latency after migrating from RDS proxy

Describe the bug
we are currently using AWS RDS proxy with IAM authentication. We implement proxy mainly for failover, IAM authentication reasons.
Now after the release of new advanced jdbc wrapper we got all features inside the advanced jdbc wrapper library itself. so we want to implement this library and directly use RDS cluster with IAM authentication and failover support. we are using spring boot (2.1.8) and using hikari pool using this example: https://github.com/aws/aws-advanced-jdbc-wrapper/tree/main/examples/SpringBootHikariExample .
we observed that instance which was directly using advanced-jdbc-wrapper library was consuming lot of CPU and latency of one of the api was more than 6 seconds and same api was taking less than 1.5 second in other instance which was using AWS RDS proxy.
That api loads lot of data in memory (200 KB) and in case of parallel requests it was able to handle with proxy implementation.  But same thing is failing with advanced-jdbc-wrapper.
Expected Behavior
Ideally there should be very minor difference in latency and CPU utilization.
CPU utilization without jar - 30%
CPU utilization after jar - 60%
API latency could have been near by upto 2 seconds as compared to proxy.
What plugins are used? What other connection properties were set?
readWriteSplitting,failover,efm2,iam
Current Behavior
CPU utilization without jar - 30%
CPU utilization after jar - 60%
Latency for memory intensive apis are more than 6 seconds as compared to 1.5 seconds(without wrapper).
Reproduction Steps

Create 2 API which is fetching lot of data from database atleast (200 MB).
Implement aws-advanced-jdbc wrapper library in spring boot (with 25 pool size)
Implement AWS RDS proxy with hikari pool (with 25 pool size)
Call those apis concurrently.
Observe the CPU and memory utilization and also see the load in RDS cluster
With proxy it is able to handle load upto some extent but with wrapper library it will start failing since beginning.

Possible Solution
No response
Additional Information/Context
No response
The AWS Advanced JDBC Driver version used
2.3.6
JDK version used
11
Operating System and version
UBUNTU 22

Hi @avnish-choudhary  ,
Thanks for reaching out and raising this issue.
Would you give more details about your application:

What connection string do you use to connect to your Hikari CP? Could you provide configuration file for your data source(s)?

Could you provide driver logs representing the issue? It's important to verify internal driver workflows and, possibly, to identify bad performant areas.

Would you be able to provde a sample app that replicate the issue?

How important for your application to use readWriteSplitting plugin?

Did you have a chance to check out other examples? For instance, the following two samples could meet your requirements.
https://github.com/aws/aws-advanced-jdbc-wrapper/tree/main/examples/SpringHibernateBalancedReaderOneDataSourceExample
and
https://github.com/aws/aws-advanced-jdbc-wrapper/tree/main/examples/SpringHibernateBalancedReaderTwoDataSourceExample

Did you try to turn on the driver telemetry? That would give you driver insights and narrow down investigation.
https://github.com/aws/aws-advanced-jdbc-wrapper/blob/main/docs/using-the-jdbc-driver/Telemetry.md

We'll take a look at this and keep you updated as we investigate. Thank you for your patience!

Hi @avnish-choudhary ,
Thanks for reaching out and raising this issue.
Would you give more details about your application:

What connection string do you use to connect to your Hikari CP? Could you provide configuration file for your data source(s)?
Could you provide driver logs representing the issue? It's important to verify internal driver workflows and, possibly, to identify bad performant areas.
Would you be able to provde a sample app that replicate the issue?
How important for your application to use readWriteSplitting plugin?
Did you have a chance to check out other examples? For instance, the following two samples could meet your requirements.
https://github.com/aws/aws-advanced-jdbc-wrapper/tree/main/examples/SpringHibernateBalancedReaderOneDataSourceExample
and
https://github.com/aws/aws-advanced-jdbc-wrapper/tree/main/examples/SpringHibernateBalancedReaderTwoDataSourceExample
Did you try to turn on the driver telemetry? That would give you driver insights and narrow down investigation.
https://github.com/aws/aws-advanced-jdbc-wrapper/blob/main/docs/using-the-jdbc-driver/Telemetry.md

We'll take a look at this and keep you updated as we investigate. Thank you for your patience!

we are using these wrapper plugins : readWriteSplitting,failover,efm2,iam
connection string: jdbc:aws-wrapper:postgresql://cluster-url:5432/testdb?currentSchema=example&ApplicationName=example&statement_timeout=5min
I tried both of the example and I found that its not supporting IAM authentication with internal pool (SF_F0).
is it a best way to use hikari with two datasource for reader and writer ?

Hi @avnish-choudhary, we have seen an issue describing a similar performance degradation in the past. Do you see the same issue with IAM disabled and are you also using a Postgres RDS instance?
To answer your question, yes we recommend setting up two datasources and disabling the readWriteSplitting plugin when using an external pool for reasons discussed here.
Could you please elaborate more on how internal pool does not work with IAM authentication? Were there connection errors? While the profile provides a set of predefined properties, client application can still override or add properties through the connection string: &wrapperPlugins=initialConnection,auroraConnectionTracker,failover,efm2,iam&iamRegion=us-west-2.

Hi @avnish-choudhary, I wanted to check in and see if you saw our response above. Please let us know. If there are no further updates on this ticket in the next few days, we will close this ticket. However, if you require further assistance, you can reopen this issue, or create a new one.
Thank you!


invalid strftime in aws_srp.py

https://github.com/capless/warrant/blob/master/warrant/aws_srp.py#L158
I think you want "%a %b %d %H:%M:%S UTC %Y" for the strftime %-d is invalid.

I believe you are correct.

It will not authenticate with %d, as it will have a leading zero.
@stephenoneal what python version and OS do you use?
http://stackoverflow.com/questions/904928/python-strftime-date-without-leading-0

python 3.4, Windows

oddly i changed it to datetime.datetime.utcnow().strftime("%a %b %d %H:%M:%S UTC %Y") locally and it's passing the PASSWORD_VERIFIER challenge

oddly i changed it to datetime.datetime.utcnow().strftime("%a %b %d %H:%M:%S UTC %Y") locally and it's passing the PASSWORD_VERIFIER challenge

It will fail on May 1.

Oh yeah, that happened at the beginning of this month.

my attention on this is not great right now.  need more coffee..
import platform
windows_os = any(platform.win32_ver())
    timestamp = test_timestamp or \
                ( datetime.datetime.utcnow().strftime("%a %b %#d %H:%M:%S UTC %Y") if windows_os \
                else datetime.datetime.utcnow().strftime("%a %b %-d %H:%M:%S UTC %Y") )

@armicron I think @stephenoneal is proposing a way to detect the OS and format the string accordingly.

I can't test it on windows.
import sys

if sys.platform.startswith('win32'):
    timestamp = test_timestamp or datetime.datetime.utcnow().strftime("%a %b %#d %H:%M:%S UTC %Y")
else:
    timestamp = test_timestamp or datetime.datetime.utcnow().strftime("%a %b %-d %H:%M:%S UTC %Y")

Fixed in #28