:latest causes kubelet to never find image

When I attempt to launch my private container image with the latest tag, the kubelet is stuck in a loop unable to pull it. If I remove the "latest" tag and resubmit the pod it launches without issue. I have 10s of other containers (probably 35?) running this same image with different names (embassy-XXX) but this specific container name always hangs up.
Here are the logs from two different minions. I have the same issue on 0.5.4, 0.7.2, and 0.8.0. I am running CoreOS on AWS using the Getting Started Guide (https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides/coreos)
Jan 07 17:59:42 ip-10-3-0-41.ec2.internal kubelet[6271]: E0107 17:59:42.464508    6271 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 17:59:42 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:42.464557    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 17:59:52 ip-10-3-0-41.ec2.internal kubelet[6271]: E0107 17:59:52.507701    6271 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 17:59:52 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:52.507749    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.908865    6271 logs.go:39] etcd DEBUG: recv.success.http://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1743
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909332    6271 logs.go:39] etcd DEBUG: get /registry/nodes/<IP_ADDRESS>/boundpods [http://<IP_ADDRESS>:4001]
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909370    6271 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909381    6271 logs.go:39] etcd DEBUG: send.request.to http://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819 | method GET
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.940102    6271 logs.go:39] etcd DEBUG: recv.response.fromhttp://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.977047    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"implicitly required container net"}): status: 'terminated', reason: 'killing' Killing 2c3b135ea85b60aac39982c4d9ee7204af7c4dd725bbe876b269df9cbbf32d07 - /k8s_net.bf618e0a_embassy-471.default.etcd_bf102b22-9696-11e4-bd40-12cd31da7477_a968ab21

Jan 07 18:00:02 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:02.876047    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:02 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:02.876105    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:12 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:12.750825    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:12 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:12.750891    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:22 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:22.755916    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:22 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:22.756023    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:32 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:32.778982    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:32 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:32.779057    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:42 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:42.750581    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:42 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:42.750678    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:52 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:52.729513    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:52 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:52.729579    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:02 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:02.733935    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:02 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:02.734025    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:12 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:12.922655    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:12 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:12.922737    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:22 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:22.749725    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:22 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:22.749783    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest

What is your image pull policy?  Can you try setting it to PullAlways and
see if that changes things?
Thanks
--brendan
On Wed, Jan 7, 2015 at 4:54 PM, W. Andrew Loe III<EMAIL_ADDRESS>wrote:

When I attempt to launch my private container image with the latest tag,
the kubelet is stuck in a loop unable to pull it. If I remove the "latest"
tag and resubmit the pod it launches without issue. I have 10s of other
containers (probably 35?) running this same image with different names
(embassy-XXX) but this specific container name always hangs up.
Here are the logs from two different minions. I have the same issue on
0.5.4, 0.7.2, and 0.8.0. I am running CoreOS on AWS using the Getting
Started Guide (
https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides/coreos
)
Jan 07 17:59:42 ip-10-3-0-41.ec2.internal kubelet[6271]: E0107 17:59:42.464508    6271 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 17:59:42 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:42.464557    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 17:59:52 ip-10-3-0-41.ec2.internal kubelet[6271]: E0107 17:59:52.507701    6271 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 17:59:52 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:52.507749    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.908865    6271 logs.go:39] etcd DEBUG: recv.success.http://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1743
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909332    6271 logs.go:39] etcd DEBUG: get /registry/nodes/<IP_ADDRESS>/boundpods [http://<IP_ADDRESS>:4001]
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909370    6271 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909381    6271 logs.go:39] etcd DEBUG: send.request.to http://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819 | method GET
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.940102    6271 logs.go:39] etcd DEBUG: recv.response.fromhttp://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.977047    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"implicitly required container net"}): status: 'terminated', reason: 'killing' Killing 2c3b135ea85b60aac39982c4d9ee7204af7c4dd725bbe876b269df9cbbf32d07 - /k8s_net.bf618e0a_embassy-471.default.etcd_bf102b22-9696-11e4-bd40-12cd31da7477_a968ab21
Jan 07 18:00:02 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:02.876047    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:02 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:02.876105    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:12 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:12.750825    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:12 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:12.750891    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:22 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:22.755916    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:22 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:22.756023    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:32 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:32.778982    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:32 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:32.779057    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:42 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:42.750581    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:42 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:42.750678    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:52 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:52.729513    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:52 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:52.729579    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:02 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:02.733935    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:02 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:02.734025    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:12 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:12.922655    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:12 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:12.922737    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:22 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:22.749725    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:22 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:22.749783    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301.

I also can't see the image at
https://registry.hub.docker.com/u/onehub/embassy/
Are you using a private image?  Maybe your .dockercfg isn't set up
correctly?
Note that prior to 0.8.0 there was a bug/race in image garbage collection
that might cause this problem, but it is resolved in 0.8.0
On Wed, Jan 7, 2015 at 4:55 PM, Brendan Burns<EMAIL_ADDRESS>wrote:

What is your image pull policy?  Can you try setting it to PullAlways and
see if that changes things?
Thanks
--brendan
On Wed, Jan 7, 2015 at 4:54 PM, W. Andrew Loe III <
<EMAIL_ADDRESS>wrote:

When I attempt to launch my private container image with the latest tag,
the kubelet is stuck in a loop unable to pull it. If I remove the "latest"
tag and resubmit the pod it launches without issue. I have 10s of other
containers (probably 35?) running this same image with different names
(embassy-XXX) but this specific container name always hangs up.
Here are the logs from two different minions. I have the same issue on
0.5.4, 0.7.2, and 0.8.0. I am running CoreOS on AWS using the Getting
Started Guide (
https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides/coreos
)
Jan 07 17:59:42 ip-10-3-0-41.ec2.internal kubelet[6271]: E0107 17:59:42.464508    6271 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 17:59:42 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:42.464557    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 17:59:52 ip-10-3-0-41.ec2.internal kubelet[6271]: E0107 17:59:52.507701    6271 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 17:59:52 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:52.507749    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.908865    6271 logs.go:39] etcd DEBUG: recv.success.http://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1743
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909332    6271 logs.go:39] etcd DEBUG: get /registry/nodes/<IP_ADDRESS>/boundpods [http://<IP_ADDRESS>:4001]
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909370    6271 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.909381    6271 logs.go:39] etcd DEBUG: send.request.to http://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819 | method GET
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.940102    6271 logs.go:39] etcd DEBUG: recv.response.fromhttp://<IP_ADDRESS>:4001/v2/keys/registry/nodes/<IP_ADDRESS>/boundpods?consistent=true&wait=true&waitIndex=1819
Jan 07 17:59:54 ip-10-3-0-41.ec2.internal kubelet[6271]: I0107 17:59:54.977047    6271 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"bf102b22-9696-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"implicitly required container net"}): status: 'terminated', reason: 'killing' Killing 2c3b135ea85b60aac39982c4d9ee7204af7c4dd725bbe876b269df9cbbf32d07 - /k8s_net.bf618e0a_embassy-471.default.etcd_bf102b22-9696-11e4-bd40-12cd31da7477_a968ab21
Jan 07 18:00:02 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:02.876047    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:02 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:02.876105    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:12 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:12.750825    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:12 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:12.750891    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:22 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:22.755916    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:22 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:22.756023    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:32 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:32.778982    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:32 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:32.779057    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:42 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:42.750581    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:42 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:42.750678    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:00:52 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:00:52.729513    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:00:52 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:00:52.729579    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:02 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:02.733935    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:02 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:02.734025    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:12 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:12.922655    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:12 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:12.922737    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
Jan 07 18:01:22 ip-10-3-0-40.ec2.internal kubelet[6568]: E0107 18:01:22.749725    6568 kubelet.go:799] Failed to pull image onehub/embassy:latest: Error: image onehub/embassy:latest not found; skipping pod embassy-471.default.etcd container embassy.
Jan 07 18:01:22 ip-10-3-0-40.ec2.internal kubelet[6568]: I0107 18:01:22.749783    6568 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"00b98ff4-9697-11e4-bd40-12cd31da7477", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers[0]"}): status: 'failed', reason: 'failed' Failed to pull image onehub/embassy:latest
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301.

The image is private, my .dockercfg does work because it pulls the container successfully for all the other pods that are running.

The job I am submitting looks like this:
{
  "apiVersion": "v1beta1",
  "desiredState": {
    "manifest": {
      "containers": [
        {
        "env": [
          {
          "name": "AMBASSADOR_LOG_LEVEL",
          "value": "6"
        },
        {
          "name": "AMBASSADOR_CLIENT_ID",
          "value": "iq5o5qoyc9t8z49fefztu0pc49gi68z"
        },
        {
          "name": "AMBASSADOR_SECRET",
          "value": "kotn3mfera0o8teoo3kenb0yqnblb0t"
        },
        {
          "name": "AMBASSADOR_SITE",
          "value": "https://ws-api.onehubmaster.com"
        },
        {
          "name": "AMBASSADOR_TOKEN",
          "value": "3zinuorl9dub19t6uw8hgkfe3jbxomf"
        },
        {
          "name": "AMBASSADOR_WORKSPACE_ID",
          "value": "471"
        },
        {
          "name": "AMBASSADOR_BROKER",
          "value"<EMAIL_ADDRESS>        },
        {
          "name": "BTSYNC_KEY",
          "value": "AFNPZS5WEG4BTPG3YAADRAE7QNXD7VBVM"
        },
        {
          "name": "BTSYNC_PORT",
          "value": "10383"
        },
        {
          "name": "ATTACHE_INOTIFY_POLL",
          "value": "1000"
        },
        {
          "name": "ATTACHE_LOG_LEVEL",
          "value": "5"
        }
        ],
        "image": "onehub/embassy",
        "name": "embassy",
        "ports": [
          {
          "containerPort": 10383,
          "hostPort": 10383
        }
        ]
      }
      ],
      "id": "embassy-471",
      "version": "v1beta1"
    }
  },
  "id": "embassy-471",
  "kind": "Pod",
  "labels": {
    "environment": "master",
    "workspace-id": "471",
    "btsync-port": "10383"
  }
}

The pod looks like this:
{
    "apiVersion": "v1beta1",
    "desiredState": {
        "manifest": {
            "containers": [
                {
                    "env": [
                        {
                            "name": "AMBASSADOR_LOG_LEVEL",
                            "value": "6"
                        },
                        {
                            "name": "AMBASSADOR_WORKSPACE_ID",
                            "value": "471"
                        }
                    ],
                    "image": "onehub/embassy",
                    "name": "embassy",
                    "ports": [
                        {
                            "containerPort": 10383,
                            "hostPort": 10383
                        }
                    ]
                }
            ],
            "id": "embassy-471",
            "version": "v1beta1"
        }
    },
    "id": "embassy-471",
    "kind": "Pod",
    "labels": {
        "environment": "master",
        "workspace-id": "471",
        "btsync-port": "10383"
    }
}

I will try with updated image pull policy.

Does not run, it says it has been scheduled. This is on 0.8.0.
Name:				embassy-471
Image(s):			onehub/embassy:latest
Host:				<IP_ADDRESS>/
Labels:				btsync-port=10383,environment=master,workspace-id=471
Status:				Pending
Replication Controllers:	<none>
Events:
Time				From		SubobjectPath	Condition	Reason		Message
Thu, 08 Jan 2015 01:09:53 +0000	{scheduler }			Pending		scheduled	Successfully assigned embassy-471 to <IP_ADDRESS>

what happens if you do a vanilla "docker pull onehub/embassy:latest" on one
of the boxes?
--brendan
On Wed, Jan 7, 2015 at 5:12 PM, W. Andrew Loe III<EMAIL_ADDRESS>wrote:

Does not run, it says it has been scheduled. This is on 0.8.0.
Name:               embassy-471
Image(s):           onehub/embassy:latest
Host:               <IP_ADDRESS>/
Labels:             btsync-port=10383,environment=master,workspace-id=471
Status:             Pending
Replication Controllers:
Events:
Time                From        SubobjectPath   Condition   Reason      Message
Thu, 08 Jan 2015 01:09:53 +0000 {scheduler }            Pending     scheduled   Successfully assigned embassy-471 to <IP_ADDRESS>
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301#issuecomment-69121252
.

Pulls it with no problem.
core@ip-10-3-0-185 ~ $ sudo docker pull onehub/embassy:latest
Pulling repository onehub/embassy
778f14d8694c: Pulling dependent layers
...
01d11c76c20f: Download complete
Status: Image is up to date for onehub/embassy:latest
core@ip-10-3-0-185 ~ $

My kubelet runs as root and my dockercfg is in /root/.dockercfg

The strangest thing is it is for only this "name", I have a bunch of other pods with the exact same image only different environment variables and name running with no problem.

core@ip-10-3-0-119 ~ $ kubectl get pods
POD                 CONTAINER(S)        IMAGE(S)                HOST                LABELS                                 STATUS
embassy-261         embassy             onehub/embassy:latest   <IP_ADDRESS>/         environment=master,workspace-id=261    Running
embassy-471         embassy             onehub/embassy          <IP_ADDRESS>/         environment=master,workspace-id=471    Running
embassy-3491        embassy             onehub/embassy:latest   <IP_ADDRESS>/         environment=master,workspace-id=3491   Running
embassy-101         embassy             onehub/embassy:latest   <IP_ADDRESS>/         environment=master,workspace-id=101    Running
embassy-3271        embassy             onehub/embassy:latest   <IP_ADDRESS>/         environment=master,workspace-id=3271   Running
embassy-3411        embassy             onehub/embassy:latest   <IP_ADDRESS>/         environment=master,workspace-id=3411   Running

471 is now running because I changed the image name.

So my bet is that the image is somehow "stuck" in a state where Docker
thinks it's been pulled, but it is not actually fully pulled.  And so
because we have an image policy of PullIfNotPresent by default, it never
gets pulled again.  That's a guess, though.
That would correlate w/ the change of name working too.
We should probably try re-pulling an image if we get a "not found" error on
container start...
--brendan
On Wed, Jan 7, 2015 at 5:19 PM, W. Andrew Loe III<EMAIL_ADDRESS>wrote:

The strangest thing is it is for only this "name", I have a bunch of other
pods with the exact same image only different environment variables and
name running with no problem.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301#issuecomment-69121941
.

But why would it only get stuck for this pod, and reliably this pod? I can start all my other ones with no problem? And if I change the imagePullPolicy to PullAlways, it still doesn't work for this pod.

Hrm, I agree that doesn't make sense...  I think that one of the layers in
the image is missing, on that particular machine, but PullAlways should
force a repull no matter what, maybe your syntax for pull always is wrong
in the pod spec?
On Jan 7, 2015 5:26 PM, "W. Andrew Loe III"<EMAIL_ADDRESS>wrote:

But why would it only get stuck for this pod, and reliably this pod? I can
start all my other ones with no problem? And if I change the
imagePullPolicy to PullAlways, it still doesn't work for this pod.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301#issuecomment-69122534
.

I used:
{
  "imagePullPolicy": "PullAlways"
}

Wouldn't the API server reject my pod if it had invalid syntax?

That appears to be valid syntax - you can triple check by doint
cluster/kubectl.sh get -o json pod [name]
On Wed, Jan 7, 2015 at 8:48 PM, W. Andrew Loe III<EMAIL_ADDRESS>wrote:

Wouldn't the API server reject my pod if it had invalid syntax?
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301#issuecomment-69135562
.

I am certain the syntax is correct. This particular pod always gets stuck in this state, what should I do to further debug?

I am stumped.  Is there anything in the docker log?  If you delete all
docker images and start anew?
On Tue, Jan 13, 2015 at 4:28 PM, W. Andrew Loe III<EMAIL_ADDRESS>
wrote:

I am certain the syntax is correct. This particular pod always gets stuck
in this state, what should I do to further debug?
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301#issuecomment-69848965
.

Maybe there is a race in docker?
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="GET /images/onehub/embassy:latest/json"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="+job image_inspect(onehub/embassy:latest)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="-job image_inspect(onehub/embassy:latest) = OK (0)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="POST /images/create?fromImage=onehub%2Fembassy&tag=latest"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="+job pull(onehub/embassy, latest)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: Error: image onehub/embassy:latest not found
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="-job pull(onehub/embassy, latest) = ERR (1)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal kubelet[28612]: E0114 00:33:24.293619   28612 kubelet.go:914] Failed to pull image "onehub/embassy:latest": Error: image onehub/embassy:latest not found; skipping pod "embassy-471.default.etcd" container "embassy".

Can you run docker with more verbosity (like add -D) ?
On Tue, Jan 13, 2015 at 4:34 PM, W. Andrew Loe III<EMAIL_ADDRESS>
wrote:

Maybe there is a race in docker?
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="GET /images/onehub/embassy:latest/json"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="+job image_inspect(onehub/embassy:latest)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="-job image_inspect(onehub/embassy:latest) = OK (0)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="POST /images/create?fromImage=onehub%2Fembassy&tag=latest"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="+job pull(onehub/embassy, latest)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: Error: image onehub/embassy:latest not found
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal docker[650]: time="2015-01-14T00:33:24Z" level="info" msg="-job pull(onehub/embassy, latest) = ERR (1)"
Jan 14 00:33:24 ip-10-3-0-185.ec2.internal kubelet[28612]: E0114 00:33:24.293619   28612 kubelet.go:914] Failed to pull image "onehub/embassy:latest": Error: image onehub/embassy:latest not found; skipping pod "embassy-471.default.etcd" container "embassy".
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3301#issuecomment-69849581
.

Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="GET /containers/8441bcd13ca39b3ff9eda67fc4bfed16bf896835d6749a3907eb9459e1f13f04/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="+job container_inspect(8441bcd13ca39b3ff9eda67fc4bfed16bf896835d6749a3907eb9459e1f13f04)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="-job container_inspect(8441bcd13ca39b3ff9eda67fc4bfed16bf896835d6749a3907eb9459e1f13f04) = OK (0)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="Calling GET /images/{name:.*}/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="GET /images/onehub/embassy:latest/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="+job image_inspect(onehub/embassy:latest)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="-job image_inspect(onehub/embassy:latest) = OK (0)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="GET /containers/f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="+job container_inspect(f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="-job container_inspect(f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94) = OK (0)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="+job containers()"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="Calling GET /images/{name:.*}/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="GET /images/onehub/embassy:latest/json"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="+job image_inspect(onehub/embassy:latest)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="-job image_inspect(onehub/embassy:latest) = OK (0)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="Calling POST /images/create"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="POST /images/create?fromImage=onehub%2Fembassy&tag=latest"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="+job pull(onehub/embassy, latest)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="[registry] Calling GET https://index.docker.io/v1/repositories/onehub/embassy/images"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="https://index.docker.io/v1/repositories/onehub/embassy/images -- HEADERS: map[User-Agent:[docker/1.4.1 go/go1.3.3 git-commit/5bc2ff8-dirty kernel/3.18.1 os/linux arch/amd64]]"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="debug" msg="hostDir: /etc/docker/certs.d/index.docker.io"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: Error: image onehub/embassy:latest not found
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:34Z" level="info" msg="-job pull(onehub/embassy, latest) = ERR (1)"
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal kubelet[28612]: E0114 00:43:34.463580   28612 kubelet.go:914] Failed to pull image "onehub/embassy:latest": Error: image onehub/embassy:latest not found; skipping pod "embassy-471.default.etcd" container "embassy".
Jan 14 00:43:34 ip-10-3-0-185.ec2.internal kubelet[28612]: I0114 00:43:34.463620   28612 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"050bcc19-9b84-11e4-8fa6-1248359f694e", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers{embassy}"}): status: 'failed', reason: 'failed' Failed to pull image "onehub/embassy:latest"
Jan 14 00:43:39 ip-10-3-0-185.ec2.internal kernel: docker0: port 1(veth7972cae) entered forwarding state
Jan 14 00:43:39 ip-10-3-0-185.ec2.internal kernel: docker0: port 2(vethb812cca) entered forwarding state
Jan 14 00:43:39 ip-10-3-0-185.ec2.internal kernel: docker0: port 3(veth1a939ba) entered forwarding state
Jan 14 00:43:39 ip-10-3-0-185.ec2.internal kernel: docker0: port 4(veth9da98f5) entered forwarding state
Jan 14 00:43:43 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:43Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:43:43 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:43Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:43:43 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:43Z" level="info" msg="+job containers()"
Jan 14 00:43:43 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:43Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:43:43 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:43Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:43:43 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:43:43Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:43:43 ip-10-

More:
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/bb0e015d1bd065d801c64e28ed9ac79d2e420c6e63dbe6fef7e85de8d14ceafc/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(bb0e015d1bd065d801c64e28ed9ac79d2e420c6e63dbe6fef7e85de8d14ceafc)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(bb0e015d1bd065d801c64e28ed9ac79d2e420c6e63dbe6fef7e85de8d14ceafc) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/ea6885abfa7bb05c5b9b6b468a119672e54091b066d9d4cd95db0f0e69b7d923/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(ea6885abfa7bb05c5b9b6b468a119672e54091b066d9d4cd95db0f0e69b7d923)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(ea6885abfa7bb05c5b9b6b468a119672e54091b066d9d4cd95db0f0e69b7d923) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /images/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /images/onehub/embassy:latest/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job image_inspect(onehub/embassy:latest)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job image_inspect(onehub/embassy:latest) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/1b3495a20c0163cb7c88d36a689b54a07e45cd081f38f8b822858d39f5f694cb/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(1b3495a20c0163cb7c88d36a689b54a07e45cd081f38f8b822858d39f5f694cb)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(1b3495a20c0163cb7c88d36a689b54a07e45cd081f38f8b822858d39f5f694cb) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/9f19cedd787f2154aa36e91ea2aa9a7e028b2e7169a7ab92fa181513b2417357/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(9f19cedd787f2154aa36e91ea2aa9a7e028b2e7169a7ab92fa181513b2417357)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(9f19cedd787f2154aa36e91ea2aa9a7e028b2e7169a7ab92fa181513b2417357) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/ea6885abfa7bb05c5b9b6b468a119672e54091b066d9d4cd95db0f0e69b7d923/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(ea6885abfa7bb05c5b9b6b468a119672e54091b066d9d4cd95db0f0e69b7d923)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(ea6885abfa7bb05c5b9b6b468a119672e54091b066d9d4cd95db0f0e69b7d923) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/bb0e015d1bd065d801c64e28ed9ac79d2e420c6e63dbe6fef7e85de8d14ceafc/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(bb0e015d1bd065d801c64e28ed9ac79d2e420c6e63dbe6fef7e85de8d14ceafc)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(bb0e015d1bd065d801c64e28ed9ac79d2e420c6e63dbe6fef7e85de8d14ceafc) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/a05447e3eed3b576e7f2ef71872122613f2ad5158f548000828ed8856214143d/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(a05447e3eed3b576e7f2ef71872122613f2ad5158f548000828ed8856214143d)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(a05447e3eed3b576e7f2ef71872122613f2ad5158f548000828ed8856214143d) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/8441bcd13ca39b3ff9eda67fc4bfed16bf896835d6749a3907eb9459e1f13f04/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(8441bcd13ca39b3ff9eda67fc4bfed16bf896835d6749a3907eb9459e1f13f04)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(8441bcd13ca39b3ff9eda67fc4bfed16bf896835d6749a3907eb9459e1f13f04) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /images/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /images/onehub/embassy:latest/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job image_inspect(onehub/embassy:latest)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job image_inspect(onehub/embassy:latest) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job container_inspect(f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job container_inspect(f20c942e20adc68d5f12801391bd397865858b14290f8adf45bc04882d3e9f94) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /containers/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /containers/json?all=1"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job containers()"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job containers() = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling GET /images/{name:.*}/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="GET /images/onehub/embassy:latest/json"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job image_inspect(onehub/embassy:latest)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job image_inspect(onehub/embassy:latest) = OK (0)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="Calling POST /images/create"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="POST /images/create?fromImage=onehub%2Fembassy&tag=latest"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="+job pull(onehub/embassy, latest)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="[registry] Calling GET https://index.docker.io/v1/repositories/onehub/embassy/images"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="https://index.docker.io/v1/repositories/onehub/embassy/images -- HEADERS: map[User-Agent:[docker/1.4.1 go/go1.3.3 git-commit/5bc2ff8-dirty kernel/3.18.1 os/linux arch/amd64]]"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="debug" msg="hostDir: /etc/docker/certs.d/index.docker.io"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: Error: image onehub/embassy:latest not found
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal docker[30940]: time="2015-01-14T00:59:14Z" level="info" msg="-job pull(onehub/embassy, latest) = ERR (1)"
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal kubelet[28612]: E0114 00:59:14.737280   28612 kubelet.go:914] Failed to pull image "onehub/embassy:latest": Error: image onehub/embassy:latest not found; skipping pod "embassy-471.default.etcd" container "embassy".
Jan 14 00:59:14 ip-10-3-0-185.ec2.internal kubelet[28612]: I0114 00:59:14.737318   28612 event.go:92] Event(api.ObjectReference{Kind:"BoundPod", Namespace:"default", Name:"embassy-471", UID:"050bcc19-9b84-11e4-8fa6-1248359f694e", APIVersion:"v1beta1", ResourceVersion:"", FieldPath:"spec.containers{embassy}"}): status: 'failed', reason: 'failed' Failed to pull image "onehub/embassy:latest"

Closing, I believe this is a problem with docker, not kubernetes.


[Web] Focus should register onDidGainAccessibilityFocus on Web only to reflect focus in Flutter

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Use case
On Flutter Web only, with accessibility enabled, the screen reader can navigate directly around the semantics DOM. When it navigates to a focusable control (e.g. a text field or button), it automatically moves focus into it. However, in these cases, the focus isn't automatically reflect in Flutter even though the screen reader explicitly chose to move focus into semantics DOM.
cc: @yjbanov @chunhtai with whom I've discussed this a bit.
Proposal
On most platforms, the callback provided to Semantics(onDidGainAccessibilityFocus: ...) is invoked as the screen reader navigates to the corresponding semantics node. However, on Web, it's not possible to detect the screen reader cursor location. In Flutter-web, this callback is invoked only if the screen reader chooses to focus the corresponding DOM element. So this is the case where the Web screen reader is deliberately choosing to move focus onto the DOM element. However, the focus is then not automatically reflected in Flutter.
The fix here is to add onDidGainAccessibilityFocus: kIsWeb ? focusNode.requestFocus : null in Focus (right here).

Duplicate of #83809

Turns out, that issue mentioned a typo-ed version didGainAccessibilityFocus, so my search-fu had failed with onDidGainAccessibilityFocus. :)


Fix a false positive for Style/TrailingBodyOnMethodDefinition

This PR fixes a false positive for Style/TrailingBodyOnMethodDefinition when endless method definition body is after newline in opening parenthesis.

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[ ] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Probably we should also add some examples with endless methods to the cop's description, but that's unrelated to the fix itself.


ARROW-6180: [C++][Parquet] Add RandomAccessFile::GetStream that returns InputStream that reads a file segment independent of the file's state, fix concurrent buffered Parquet column reads

This enables different functions to read portions of a RandomAccessFile as an InputStream without interfering with each other.
This also addresses PARQUET-1636 and adds a unit test for buffered column chunk reads. In the refactor to use the Arrow IO interfaces, I broke this by allowing the raw RandomAccessFile to be passed into multiple BufferedInputStream at once, so the file position was being manipulated by different column readers. We didn't catch the problem because we didn't have any unit tests, so this patch addresses that deficiency.

cc @majetideepak @czxrrr

From a high-level POV, note that InputStreams are not supposed to be thread-safe. Abstractly, perhaps Parquet should use ReadAt. Well, anyway...

I'm removing all notions of "thread-safe" so we aren't haggling about that, it's really about "isolation" so that the columns can buffer a file segment without interfering with each other

@ursabot build

Codecov Report

Merging #5085 into master will increase coverage by 1.62%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5085      +/-   ##
==========================================
+ Coverage   87.63%   89.25%   +1.62%
==========================================
  Files        1013      731     -282
  Lines      144891   104130   -40761
  Branches     1418        0    -1418
==========================================
- Hits       126971    92944   -34027
+ Misses      17558    11186    -6372
+ Partials      362        0     -362

Impacted Files
Coverage Δ

cpp/src/parquet/properties.h
97.67% <ø> (+0.11%)
:arrow_up:

cpp/src/arrow/io/interfaces.h
90% <ø> (ø)
:arrow_up:

cpp/src/arrow/testing/random.h
100% <ø> (ø)
:arrow_up:

cpp/src/parquet/properties.cc
100% <100%> (+26.66%)
:arrow_up:

cpp/src/arrow/io/memory-test.cc
100% <100%> (ø)
:arrow_up:

cpp/src/arrow/io/interfaces.cc
82.69% <100%> (+27.69%)
:arrow_up:

cpp/src/parquet/reader-test.cc
100% <100%> (ø)
:arrow_up:

cpp/src/arrow/csv/column-builder.cc
95.54% <0%> (-1.49%)
:arrow_down:

js/src/util/fn.ts

go/arrow/memory/memory_avx2_amd64.go

... and 284 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cef7e80...5da050c. Read the comment docs.

Rebased. I'll merge this once the builds run if there are no more comments

Travis CI: https://travis-ci.org/wesm/arrow/builds/572344045
Appveyor: https://ci.appveyor.com/project/wesm/arrow/builds/26721644


Use jotai to store global preferences

Playwright tests will cover these changes.

 2.0 FE: Replace GlobalPreferencesContext

please update this ADR with this PR


My Documents and Documents Bug

I have two 1909 PC's with the latest version of OneDrive. While using silent KFM via GPO. When i login to the fist PC, it creates the 'Desktop', 'Documents' and 'Pictures' folders within onedrive and redirects my folders to them.
However when i login to another PC, The desktop stays the same but instead of redirecting my documents to the already created' documents' folder within onedrive, it creates a 'My Documents' folder in OneDrive and redirects to that instead. The same happens for the Pictures folder. So I end up with this:

Am i doing something wrong or is this a bug?

@avidpontoon thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


chore(*): bump CoreOS to 681.1.0

See https://coreos.com/releases/#681.1.0

Hold the phone...https://coreos.com/releases/#681.2.0. I'll update this PR.

Code LGTM

LGTM.


Info about upcoming removal of packages in the General registry

As described in https://discourse.julialang.org/t/ann-plans-for-removing-packages-that-do-not-yet-support-1-0-from-the-general-registry/ we are planning on removing packages that do not support 1.0 from the General registry. This package has been detected to not support 1.0 and is thus slated to be removed. The removal of packages from the registry will happen approximately a month after this issue is open.
To transition to the new Pkg system using Project.toml, see https://github.com/JuliaRegistries/Registrator.jl#transitioning-from-require-to-projecttoml.
To then tag a new version of the package, see https://github.com/JuliaRegistries/Registrator.jl#via-the-github-app.
If you believe this package has erroneously been detected as not supporting 1.0 or have any other questions, don't hesitate to discuss it here or in the thread linked at the top of this post.

@KristofferC Thanks for the notification -- we are in the progress of addressing this, but haven't been able to set aside time to wrap the final steps yet.