Update k8s libraries to 1.12.3

This PR updates k8s libraries to 1.12.3

This change is

Travis tests have failed
Hey @johnugeorge,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     go build -o tf-operator.v1beta2 github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2

# github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/transport
vendor/k8s.io/client-go/transport/round_trippers.go:437:9: undefined: strings.Builder

     gometalinter --config=linter_config.json --vendor ./...

WARNING: failed to make error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go a relative path: lstat error: failed to check packages: errors while loading package github.com: no such file or directory
vendor/k8s.io/client-go/transport/round_trippers.go:437:9:warning: Builder not declared by package strings (unconvert)
cmd/tf-operator.v1beta2/app/server.go:158:22:warning: unused variable or constant cannot use run (variable of type func(<-chan struct{})) as func(context.Context) value in struct literal (varcheck)
cmd/tf-operator.v1beta2/app/server.go:152:20:warning: unused variable or constant cannot use (election.LeaderElectionConfig literal) (value of type github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/leaderelection.LeaderElectionConfig) as context.Context value in argument to election.RunOrDie: missing method Deadline (varcheck)
error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:158:22:warning: error return value not checked (cannot use run (variable of type func(<-chan struct{})) as func(context.Context) value in struct literal /home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:152:20: cannot use (election.LeaderElectionConfig literal) (value of type github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/leaderelection.LeaderElectionConfig) as context.Context value in argument to election.RunOrDie: missing method Deadline /home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:163:3: too few arguments in call to election.RunOrDie]) (errcheck)
cmd/tf-operator.v1beta2/app/server.go:158:22:error: cannot use run (variable of type func(<-chan struct{})) as func(context.Context) value in struct literal (gotype, unconvert)
cmd/tf-operator.v1beta2/app/server.go:152:20:error: cannot use (election.LeaderElectionConfig literal) (value of type github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/leaderelection.LeaderElectionConfig) as context.Context value in argument to election.RunOrDie: missing method Deadline (gotype, unconvert)
cmd/tf-operator.v1beta2/app/server.go:163:3:error: too few arguments in call to election.RunOrDie (gotype, unconvert)
vendor/k8s.io/client-go/transport/round_trippers.go:437:9:warning: unused variable or constant Builder not declared by package strings (varcheck)
cmd/tf-operator.v1beta2/app/server.go:163:3:warning: unused variable or constant too few arguments in call to election.RunOrDie (varcheck)

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
FAIL	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1 [build failed]
exit status 2: warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/jobcontroller
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/control
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/logger
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util/k8sutil
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util/signals
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util/train
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/version
# github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/transport
vendor/k8s.io/client-go/transport/round_trippers.go:437:9: undefined: strings.Builder

2nd Build
View build log

     go build -o tf-operator.v1beta2 github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2

# github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app
cmd/tf-operator.v1beta2/app/server.go:152:19: not enough arguments in call to leaderelection.RunOrDie
	have (leaderelection.LeaderElectionConfig)
	want (context.Context, leaderelection.LeaderElectionConfig)
cmd/tf-operator.v1beta2/app/server.go:158:20: cannot use run (type func(<-chan struct {})) as type func(context.Context) in field value
travis_time:end:14037b88:start=1564666204243691028,finish=1564666271481665769,duration=67237974741

     gometalinter --config=linter_config.json --vendor ./...

WARNING: failed to make error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go a relative path: lstat error: failed to check packages: errors while loading package github.com: no such file or directory
error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:158:22:warning: error return value not checked (cannot use run (variable of type func(<-chan struct{})) as func(context.Context) value in struct literal /home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:152:20: cannot use (election.LeaderElectionConfig literal) (value of type github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/leaderelection.LeaderElectionConfig) as context.Context value in argument to election.RunOrDie: missing method Deadline /home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2/app/server.go:163:3: too few arguments in call to election.RunOrDie]) (errcheck)
cmd/tf-operator.v1beta2/app/server.go:163:3:warning: too few arguments in call to election.RunOrDie (compile) (staticcheck, staticcheck)
cmd/tf-operator.v1beta2/app/server.go:152:19:error: not enough arguments in call to leaderelection.RunOrDie (vet)
cmd/tf-operator.v1beta2/app/server.go:158:20:error: cannot use run (type func(<-chan struct {})) as type func(context.Context) in field value (vet)
cmd/tf-operator.v1beta2/app/server.go:158:22:error: cannot use run (variable of type func(<-chan struct{})) as func(context.Context) value in struct literal (gotype, unconvert)
cmd/tf-operator.v1beta2/app/server.go:152:20:error: cannot use (election.LeaderElectionConfig literal) (value of type github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/leaderelection.LeaderElectionConfig) as context.Context value in argument to election.RunOrDie: missing method Deadline (gotype, unconvert)
cmd/tf-operator.v1beta2/app/server.go:163:3:error: too few arguments in call to election.RunOrDie (gotype, unconvert)
cmd/tf-operator.v1beta2/app/server.go:158:22:warning: unused variable or constant cannot use run (variable of type func(<-chan struct{})) as func(context.Context) value in struct literal (varcheck)
cmd/tf-operator.v1beta2/app/server.go:152:20:warning: unused variable or constant cannot use (election.LeaderElectionConfig literal) (value of type github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/leaderelection.LeaderElectionConfig) as context.Context value in argument to election.RunOrDie: missing method Deadline (varcheck)
cmd/tf-operator.v1beta2/app/server.go:163:3:warning: unused variable or constant too few arguments in call to election.RunOrDie (varcheck)
cmd/tf-operator.v1beta2/app/server.go:158:22:warning: cannot use run (variable of type func(<-chan struct{})) as func(context.Context) value in struct literal (compile) (staticcheck, staticcheck)
cmd/tf-operator.v1beta2/app/server.go:152:20:warning: cannot use (election.LeaderElectionConfig literal) (value of type github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/tools/leaderelection.LeaderElectionConfig) as context.Context value in argument to election.RunOrDie: missing method Deadline (compile) (staticcheck, staticcheck)
travis_time:end:17a55848:start=1564666282018695548,finish=1564666431988388737,duration=149969693189

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.029s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.028s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-01T13:34:07Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-01T13:34:07Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-01T13:34:07Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-01T13:34:07Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.030s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.019s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-01T13:34:25Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-01T13:34:25Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.056s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.499186ms)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.23108ms)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (620.43µs)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.375626ms)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0801 13:34:32.295906    9623 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4204d7860), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420530dc0), "PS":(*v1.ReplicaSpec)(0xc420531080)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc42070f120)}, StartTime:(*v1.Time)(0xc4202d62e0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.400454ms)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (205.1µs)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.382617ms)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (346.121µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:32Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:32Z" level=info msg="Started workers"
time="2019-08-01T13:34:32Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:32Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:32Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-01T13:34:32Z" level=info msg="Shutting down workers"
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (494.999µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-01T13:34:32Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:32Z" level=info msg="Started workers"
time="2019-08-01T13:34:32Z" level=info msg="Shutting down workers"
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (257.896µs)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (352.864µs)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (540.406µs)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (272.672µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (222.281µs)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:32Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:32Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (191.222µs)" job=default.test-tfjob
time="2019-08-01T13:34:32Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:32Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:34Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:34Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:34Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:34Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:34Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:34Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (646.101µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-01T13:34:34Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:34Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:34Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:34Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:36Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-01T13:34:36Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:36Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:36Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:36Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-01T13:34:36Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (323.864µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-01T13:34:36Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:36Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:36Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:36Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:37Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:37Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-01T13:34:37Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-01T13:34:37Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0801 13:34:37.029186    9623 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420894550), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4209b8840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc42059fec0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-01T13:34:37Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0801 13:34:37.029310    9623 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420894550), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4209b8840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc42059fec0)}, StartTime:(*v1.Time)(0xc4209bda80), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-01T13:34:37Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:37Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (532.085µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:37Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:37Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:37Z" level=info msg="Started workers"
time="2019-08-01T13:34:37Z" level=info msg="Shutting down workers"
time="2019-08-01T13:34:37Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:37Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-01T13:34:37Z" level=info msg="Shutting down workers"
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0801 13:34:37.131842    9623 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420922dc0)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420a4a3f0)}, StartTime:(*v1.Time)(0xc420a48f80), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:37Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-01T13:34:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.919s	coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.202611ms)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.051473ms)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.134667ms)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (417.181µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (765.055µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0801 13:34:43.204232    9742 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4204ecf80), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc4207a2000), "PS":(*v1beta2.ReplicaSpec)(0xc4207a22c0)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420733470)}, StartTime:(*v1.Time)(0xc4207a0720), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.61456ms)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (616.558µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.14591ms)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:43Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:43Z" level=info msg="Started workers"
time="2019-08-01T13:34:43Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:43Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:43Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-01T13:34:43Z" level=info msg="Shutting down workers"
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (818.756µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
time="2019-08-01T13:34:43Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:43Z" level=info msg="Started workers"
time="2019-08-01T13:34:43Z" level=info msg="Shutting down workers"
=== RUN   TestDeletePodsAndServices
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (493.365µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (344.847µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (596.796µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (539.628µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (256.435µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (286.278µs)" job=default.test-tfjob
time="2019-08-01T13:34:43Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:43Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:45Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:45Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:45Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:45Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:45Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:45Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-01T13:34:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (713.009µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-01T13:34:45Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:45Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-01T13:34:45Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:45Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:47Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-01T13:34:47Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:47Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:47Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:47Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-01T13:34:47Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (570.888µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-01T13:34:47Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:47Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:47Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:47Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:47Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:47Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-01T13:34:47Z" level=info msg="Shutting down workers"
time="2019-08-01T13:34:47Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:47Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:47Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:47Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:47Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-01T13:34:47Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-01T13:34:47Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0801 13:34:47.943111    9742 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4204ec330), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420550b00)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4201ed040)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-01T13:34:47Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:47Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0801 13:34:47.943848    9742 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4204ec330), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420550b00)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4201ed040)}, StartTime:(*v1.Time)(0xc4203af280), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-01T13:34:47Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-01T13:34:47Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.221392ms)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
time="2019-08-01T13:34:47Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:47Z" level=info msg="Started workers"
time="2019-08-01T13:34:47Z" level=info msg="Shutting down workers"
=== RUN   TestAddService
time="2019-08-01T13:34:47Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:47Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:47Z" level=info msg="Starting TFJob controller"
time="2019-08-01T13:34:47Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-01T13:34:48Z" level=info msg="Starting 1 workers"
time="2019-08-01T13:34:48Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-01T13:34:48Z" level=info msg="Shutting down workers"
time="2019-08-01T13:34:48Z" level=info msg="Creating TFJob controller"
time="2019-08-01T13:34:48Z" level=info msg="Creating Job controller"
time="2019-08-01T13:34:48Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0801 13:34:48.048575    9742 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc4208aa000)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4203a1970)}, StartTime:(*v1.Time)(0xc420531d60), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'

Travis tests have failed
Hey @johnugeorge,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     go build -o tf-operator.v1beta2 github.com/kubeflow/tf-operator/cmd/tf-operator.v1beta2

# github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/transport
vendor/k8s.io/client-go/transport/round_trippers.go:437:9: undefined: strings.Builder

     gometalinter --config=linter_config.json --vendor ./...

vendor/k8s.io/client-go/transport/round_trippers.go:437:9:warning: unused variable or constant Builder not declared by package strings (varcheck)
vendor/k8s.io/client-go/transport/round_trippers.go:437:9:warning: Builder not declared by package strings (unconvert)

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
FAIL	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1 [build failed]
exit status 2: warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/jobcontroller
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/control
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/logger
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util/k8sutil
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util/signals
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/util/train
warning: no packages being tested depend on github.com/kubeflow/tf-operator/pkg/version
# github.com/kubeflow/tf-operator/vendor/k8s.io/client-go/transport
vendor/k8s.io/client-go/transport/round_trippers.go:437:9: undefined: strings.Builder

TravisBuddy Request Identifier: 15728720-b463-11e9-a5f8-ad46f5ea24b0

/retest

/assign @richardsliu
/lgtm

/approve