Prometheus Monitoring for TF Operator

Metrics being targeted to track using Prometheus:

Report for each pod (tf-operator, tf-master, tf-ps and tf-worker), reports on:

[x] CPU usage
[ ] GPU usage
[x] Memory usage
[x] Network usage
[x] I/O usage
[x] Keep-Alive check
[x] Is Leader check

Report TFJob metrics:

[ ] Job creation
[ ] Job deletion
[ ] Jobs created per hour
[ ] Successful job completions

Notes:

Helm charts being used to install Prometheus monitoring components.

Not ready for review or merge

This change is

Approach:
The metrics -
CPU usage
GPU usage*
Memory usage
Network usage
I/O usage
Can be obtained by cAdvisor stats being captured in Prometheus.
For the metrics -
Is Leader check
Job creation
Job deletion
Jobs created per hour
Successful job completions
We can use go Prometheus instrumentation for reporting these metrics from the operator CRD component.
Implements issue #988

/hold wip

Coverage remained the same at 76.744% when pulling 4e2e16bba530c24af8c4b8a45e470b269ce96cb3 on krishnadurai:feature/prometheus_integration into 61f9ea6e9b677e9464e00b623856d1de39f378f8 on kubeflow:master.

Travis tests have failed
Hey @krishnadurai,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     go build -o tf-operator.v1 github.com/kubeflow/tf-operator/cmd/tf-operator.v1

cmd/tf-operator.v1/app/server.go:42:2: cannot find package "github.com/prometheus/client_golang/prometheus" in any of:
	/home/travis/gopath/src/github.com/kubeflow/tf-operator/vendor/github.com/prometheus/client_golang/prometheus (vendor tree)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/github.com/prometheus/client_golang/prometheus (from $GOROOT)
	/home/travis/gopath/src/github.com/prometheus/client_golang/prometheus (from $GOPATH)
cmd/tf-operator.v1/app/server.go:43:2: cannot find package "github.com/prometheus/client_golang/prometheus/promauto" in any of:
	/home/travis/gopath/src/github.com/kubeflow/tf-operator/vendor/github.com/prometheus/client_golang/prometheus/promauto (vendor tree)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/github.com/prometheus/client_golang/prometheus/promauto (from $GOROOT)
	/home/travis/gopath/src/github.com/prometheus/client_golang/prometheus/promauto (from $GOPATH)
cmd/tf-operator.v1/main.go:28:2: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of:
	/home/travis/gopath/src/github.com/kubeflow/tf-operator/vendor/github.com/prometheus/client_golang/prometheus/promhttp (vendor tree)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOROOT)
	/home/travis/gopath/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOPATH)

     gometalinter --config=linter_config.json --vendor ./...

WARNING: failed to make error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1/main.go a relative path: lstat error: failed to check packages: errors while loading package github.com: no such file or directory
cmd/tf-operator.v1/main.go:28:2:warning: unused variable or constant could not import github.com/prometheus/client_golang/prometheus/promhttp (cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of: (varcheck)
cmd/tf-operator.v1/main.go:1::warning: file is not goimported (goimports)
error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1/main.go:28:2:warning: error return value not checked (could not import github.com/prometheus/client_golang/prometheus/promhttp (invalid package name: "")]) (errcheck)
cmd/tf-operator.v1/main.go:26:2:error: could not import github.com/kubeflow/tf-operator/cmd/tf-operator.v1/app (type-checking package "github.com/kubeflow/tf-operator/cmd/tf-operator.v1/app" failed (/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1/app/server.go:42:2: could not import github.com/prometheus/client_golang/prometheus (cannot find package "github.com/prometheus/client_golang/prometheus" in any of: (gotype)
cmd/tf-operator.v1/main.go:28:2:error: could not import github.com/prometheus/client_golang/prometheus/promhttp (cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of: (gotype, unconvert)
cmd/tf-operator.v1/app/server.go:43:2:error: could not import github.com/prometheus/client_golang/prometheus/promauto (cannot find package "github.com/prometheus/client_golang/prometheus/promauto" in any of: (gotype, unconvert)
cmd/tf-operator.v1/app/server.go:42:2:warning: unused variable or constant could not import github.com/prometheus/client_golang/prometheus (cannot find package "github.com/prometheus/client_golang/prometheus" in any of: (varcheck)
cmd/tf-operator.v1/app/server.go:42:2:error: could not import github.com/prometheus/client_golang/prometheus (cannot find package "github.com/prometheus/client_golang/prometheus" in any of: (gotype, unconvert)
cmd/tf-operator.v1/app/server.go:43:2:warning: unused variable or constant could not import github.com/prometheus/client_golang/prometheus/promauto (cannot find package "github.com/prometheus/client_golang/prometheus/promauto" in any of: (varcheck)
cmd/tf-operator.v1/app/server.go:1::warning: file is not goimported (goimports)
cmd/tf-operator.v1/main.go:43::warning: Errors unhandled.,LOW,HIGH (gosec)

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 2.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.063s
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 2.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.047s
=== RUN   TestValidateBetaTwoTFJobSpec
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 2.8% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.046s
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 1.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.047s
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 4.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.065s
=== RUN   TestNormalPath
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
--- PASS: TestDeletePodsAndServices (0.00s)
=== RUN   TestCleanupTFJob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
--- PASS: TestStatus (0.01s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 31.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.916s
=== RUN   TestNormalPath
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
--- PASS: TestStatus (0.01s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 31.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow	4.911s
?   	github.com/kubeflow/tf-operator/pkg/logger	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/k8sutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/signals	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/train	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/version	[no test files]
ignoring pkg/apis/common/v1/openapi_generated.go
ignoring pkg/apis/common/v1/zz_generated.deepcopy.go
ignoring pkg/apis/common/v1/zz_generated.defaults.go
ignoring pkg/apis/common/v1beta2/openapi_generated.go
ignoring pkg/apis/common/v1beta2/zz_generated.deepcopy.go
ignoring pkg/apis/common/v1beta2/zz_generated.defaults.go
ignoring pkg/apis/tensorflow/v1/openapi_generated.go
ignoring pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1/zz_generated.defaults.go
ignoring pkg/apis/tensorflow/v1beta2/openapi_generated.go
ignoring pkg/apis/tensorflow/v1beta2/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1beta2/zz_generated.defaults.go
ignoring pkg/client/clientset/versioned/clientset.go
ignoring pkg/client/clientset/versioned/fake/clientset_generated.go
ignoring pkg/client/clientset/versioned/fake/register.go
ignoring pkg/client/clientset/versioned/scheme/register.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tensorflow_client.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tfjob.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1/tensorflow_client.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1/tfjob.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake/fake_tensorflow_client.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake/fake_tfjob.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1beta2/tensorflow_client.go
ignoring pkg/client/clientset/versioned/typed/tensorflow/v1beta2/tfjob.go
ignoring pkg/client/informers/externalversions/factory.go
ignoring pkg/client/informers/externalversions/generic.go
ignoring pkg/client/informers/externalversions/tensorflow/interface.go
ignoring pkg/client/informers/externalversions/tensorflow/v1/interface.go
ignoring pkg/client/informers/externalversions/tensorflow/v1/tfjob.go
ignoring pkg/client/informers/externalversions/tensorflow/v1beta2/interface.go
ignoring pkg/client/informers/externalversions/tensorflow/v1beta2/tfjob.go
ignoring pkg/client/listers/tensorflow/v1/tfjob.go
ignoring pkg/client/listers/tensorflow/v1beta2/tfjob.go
Job #2448.1
https://coveralls.io/jobs/49006346

2nd Build
View build log

     go build -o tf-operator.v1 github.com/kubeflow/tf-operator/cmd/tf-operator.v1

cmd/tf-operator.v1/app/server.go:42:2: cannot find package "github.com/prometheus/client_golang/prometheus" in any of:
	/home/travis/gopath/src/github.com/kubeflow/tf-operator/vendor/github.com/prometheus/client_golang/prometheus (vendor tree)
	/home/travis/.gimme/versions/go1.10.linux.amd64/src/github.com/prometheus/client_golang/prometheus (from $GOROOT)
	/home/travis/gopath/src/github.com/prometheus/client_golang/prometheus (from $GOPATH)
cmd/tf-operator.v1/app/server.go:43:2: cannot find package "github.com/prometheus/client_golang/prometheus/promauto" in any of:
	/home/travis/gopath/src/github.com/kubeflow/tf-operator/vendor/github.com/prometheus/client_golang/prometheus/promauto (vendor tree)
	/home/travis/.gimme/versions/go1.10.linux.amd64/src/github.com/prometheus/client_golang/prometheus/promauto (from $GOROOT)
	/home/travis/gopath/src/github.com/prometheus/client_golang/prometheus/promauto (from $GOPATH)
cmd/tf-operator.v1/main.go:28:2: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of:
	/home/travis/gopath/src/github.com/kubeflow/tf-operator/vendor/github.com/prometheus/client_golang/prometheus/promhttp (vendor tree)
	/home/travis/.gimme/versions/go1.10.linux.amd64/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOROOT)
	/home/travis/gopath/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOPATH)
travis_time:end:04ee7d0e:start=1559061312743953238,finish=1559061313066642257,duration=322689019

     gometalinter --config=linter_config.json --vendor ./...

WARNING: failed to make error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1/main.go a relative path: lstat error: failed to check packages: errors while loading package github.com: no such file or directory
cmd/tf-operator.v1/app/server.go:42:2:error: cannot find package "github.com/prometheus/client_golang/prometheus" in any of: (vet)
cmd/tf-operator.v1/app/server.go:43:2:error: cannot find package "github.com/prometheus/client_golang/prometheus/promauto" in any of: (vet)
cmd/tf-operator.v1/main.go:1::warning: file is not goimported (goimports)
cmd/tf-operator.v1/app/server.go:42:2:warning: unused variable or constant could not import github.com/prometheus/client_golang/prometheus (cannot find package "github.com/prometheus/client_golang/prometheus" in any of: (varcheck)
cmd/tf-operator.v1/main.go:28:2:warning: could not import github.com/prometheus/client_golang/prometheus/promhttp (invalid package name: "") (compile) (staticcheck)
cmd/tf-operator.v1/app/server.go:43:2:warning: could not import github.com/prometheus/client_golang/prometheus/promauto (invalid package name: "") (compile) (staticcheck)
cmd/tf-operator.v1/main.go:26:2:error: could not import github.com/kubeflow/tf-operator/cmd/tf-operator.v1/app (type-checking package "github.com/kubeflow/tf-operator/cmd/tf-operator.v1/app" failed (/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1/app/server.go:42:2: could not import github.com/prometheus/client_golang/prometheus (cannot find package "github.com/prometheus/client_golang/prometheus" in any of: (gotype)
error: failed to check packages: errors while loading package github.com/kubeflow/tf-operator/cmd/tf-operator.v1: [/home/travis/gopath/src/github.com/kubeflow/tf-operator/cmd/tf-operator.v1/main.go:28:2:warning: error return value not checked (could not import github.com/prometheus/client_golang/prometheus/promhttp (invalid package name: "")]) (errcheck)
cmd/tf-operator.v1/main.go:43::warning: Errors unhandled.,LOW,HIGH (gosec)
cmd/tf-operator.v1/main.go:28:2:error: could not import github.com/prometheus/client_golang/prometheus/promhttp (cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of: (gotype, unconvert)
cmd/tf-operator.v1/app/server.go:42:2:error: could not import github.com/prometheus/client_golang/prometheus (cannot find package "github.com/prometheus/client_golang/prometheus" in any of: (gotype, unconvert)
cmd/tf-operator.v1/app/server.go:43:2:error: could not import github.com/prometheus/client_golang/prometheus/promauto (cannot find package "github.com/prometheus/client_golang/prometheus/promauto" in any of: (gotype, unconvert)
cmd/tf-operator.v1/main.go:28:2:error: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of: (vet)
cmd/tf-operator.v1/app/server.go:1::warning: file is not goimported (goimports)
cmd/tf-operator.v1/app/server.go:42:2:warning: could not import github.com/prometheus/client_golang/prometheus (invalid package name: "") (compile) (staticcheck)
cmd/tf-operator.v1/app/server.go:43:2:warning: unused variable or constant could not import github.com/prometheus/client_golang/prometheus/promauto (cannot find package "github.com/prometheus/client_golang/prometheus/promauto" in any of: (varcheck)
cmd/tf-operator.v1/main.go:28:2:warning: unused variable or constant could not import github.com/prometheus/client_golang/prometheus/promhttp (cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of: (varcheck)
travis_time:end:139f70fa:start=1559061316601940773,finish=1559061435129630043,duration=118527689270

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.028s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.027s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-05-28T16:37:31Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-05-28T16:37:31Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-05-28T16:37:31Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-05-28T16:37:31Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.028s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.018s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-05-28T16:37:56Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-05-28T16:37:56Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.058s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.686341ms)" job=default.test-tfjob
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (668.54µs)" job=default.test-tfjob
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (434.759µs)" job=default.test-tfjob
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (517.149µs)" job=default.test-tfjob
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (512.09µs)" job=default.test-tfjob
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (394.716µs)" job=default.test-tfjob
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.536654ms)" job=default.test-tfjob
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:09Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:09Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:09Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:09Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:09Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0528 16:38:09.759994    9823 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4204de590), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc4201318c0), "Worker":(*v1.ReplicaSpec)(0xc4202a7340)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"PS":(*v1.ReplicaStatus)(0xc42041b4a0), "Worker":(*v1.ReplicaStatus)(0xc42041b4e0)}, StartTime:(*v1.Time)(0xc4200d64e0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-05-28T16:38:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (983.814µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
time="2019-05-28T16:38:09Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:09Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:09Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:09Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:09Z" level=info msg="Started workers"
time="2019-05-28T16:38:10Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:10Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:10Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-05-28T16:38:10Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:10Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (325.711µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-05-28T16:38:10Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:10Z" level=info msg="Started workers"
time="2019-05-28T16:38:10Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (261.232µs)" job=default.test-tfjob
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (246.131µs)" job=default.test-tfjob
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (502.485µs)" job=default.test-tfjob
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.045693ms)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (216.111µs)" job=default.test-tfjob
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (230.602µs)" job=default.test-tfjob
time="2019-05-28T16:38:10Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:10Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:12Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:12Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:12Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:12Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:12Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:12Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (499.932µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2019-05-28T16:38:12Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:12Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:12Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:12Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-05-28T16:38:14Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:14Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-05-28T16:38:14Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (328.748µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-05-28T16:38:14Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:14Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:14Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:14Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-05-28T16:38:14Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:14Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:14Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:14Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-05-28T16:38:14Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0528 16:38:14.487685    9823 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4206bbee0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420952840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4203906e0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-05-28T16:38:14Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0528 16:38:14.487814    9823 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4206bbee0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420952840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4203906e0)}, StartTime:(*v1.Time)(0xc42074e680), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-05-28T16:38:14Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:14Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (849.431µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-05-28T16:38:14Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:14Z" level=info msg="Started workers"
time="2019-05-28T16:38:14Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:14Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:14Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:14Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:14Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-05-28T16:38:14Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:14Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:14Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0528 16:38:14.591401    9823 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4209d8840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4209a3820)}, StartTime:(*v1.Time)(0xc4209e8f40), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
 Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-05-28T16:38:20Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.059226ms)" job=default.test-tfjob
time="2019-05-28T16:38:20Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:20Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:20Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (492.697µs)" job=default.test-tfjob
time="2019-05-28T16:38:20Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:20Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:20Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.362398ms)" job=default.test-tfjob
time="2019-05-28T16:38:20Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:20Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:20Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (356.489µs)" job=default.test-tfjob
time="2019-05-28T16:38:20Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:20Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:20Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (474.116µs)" job=default.test-tfjob
time="2019-05-28T16:38:20Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:20Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:20Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-05-28T16:38:20Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:20Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:20Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (527.835µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
time="2019-05-28T16:38:20Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:20Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:20Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:20Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:20Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:20Z" level=info msg="Started workers"
time="2019-05-28T16:38:21Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:21Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:21Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
time="2019-05-28T16:38:21Z" level=info msg="Shutting down workers"
=== RUN   TestCopyLabelsAndAnnotation
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (303.745µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (265.311µs)" job=default.test-tfjob
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (263.183µs)" job=default.test-tfjob
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (339.141µs)" job=default.test-tfjob
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (312.148µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.00s)
=== RUN   TestCleanupTFJob
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:21Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:21Z" level=info msg="Started workers"
time="2019-05-28T16:38:21Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (254.664µs)" job=default.test-tfjob
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (231.97µs)" job=default.test-tfjob
time="2019-05-28T16:38:21Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:21Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:23Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:23Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:23Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:23Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:23Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:23Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:23Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-05-28T16:38:23Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (832.605µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-05-28T16:38:23Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:23Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:23Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:23Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:23Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-05-28T16:38:23Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:23Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:25Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-05-28T16:38:25Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:25Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:25Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-05-28T16:38:25Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (346.79µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-05-28T16:38:25Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:25Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:25Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:25Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-05-28T16:38:25Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-05-28T16:38:25Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:25Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:25Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-05-28T16:38:25Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-05-28T16:38:25Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0528 16:38:25.253413    9927 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4200b85a0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420457600)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4205cf250)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-05-28T16:38:25Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0528 16:38:25.253729    9927 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4200b85a0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420457600)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4205cf250)}, StartTime:(*v1.Time)(0xc420310b40), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-05-28T16:38:25Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-05-28T16:38:25Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (755.561µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
time="2019-05-28T16:38:25Z" level=info msg="Starting 1 workers"
=== RUN   TestAddService
time="2019-05-28T16:38:25Z" level=info msg="Started workers"
time="2019-05-28T16:38:25Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:25Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:25Z" level=info msg="Starting TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Waiting for informer caches to sync"
time="2019-05-28T16:38:25Z" level=info msg="Starting 1 workers"
time="2019-05-28T16:38:25Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-05-28T16:38:25Z" level=info msg="Shutting down workers"
time="2019-05-28T16:38:25Z" level=info msg="Creating TFJob controller"
time="2019-05-28T16:38:25Z" level=info msg="Creating Job controller"
time="2019-05-28T16:38:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0528 16:38:25.356348    9927 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc4206442c0)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4204e2110)}, StartTime:(*v1.Time)(0xc42000ce00), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'

It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     gometalinter --config=linter_config.json --vendor ./...

cmd/tf-operator.v1/main.go:43:22:warning: error return value not checked (http.ListenAndServe(fmt.Sprintf(":%s", monitoringPort), nil)) (errcheck)
cmd/tf-operator.v1/main.go:1::warning: file is not goimported (goimports)
cmd/tf-operator.v1/app/server.go:1::warning: file is not goimported (goimports)
pkg/controller.v1/tensorflow/controller.go:1::warning: file is not goimported (goimports)
pkg/controller.v1/tensorflow/job.go:1::warning: file is not goimported (goimports)
pkg/controller.v1/tensorflow/status.go:1::warning: file is not goimported (goimports)
cmd/tf-operator.v1/main.go:43::warning: Errors unhandled.,LOW,HIGH (gosec)

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 2.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.048s
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 2.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.048s
=== RUN   TestValidateBetaTwoTFJobSpec
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 2.8% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.047s
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 1.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.047s
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 4.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.068s
=== RUN   TestNormalPath
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
--- PASS: TestStatus (0.01s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS