apiserver was unable to write a JSON response: http2: stream closed

What happened?
Deploy prometheus adapter with kube-prometheus
Got below error in prometheus adapter pod log(repeat every 30 seconds):
I0506 11:28:29.190304       1 adapter.go:93] successfully using in-cluster auth
I0506 11:28:29.624391       1 serving.go:273] Generated self-signed cert (/var/run/serving-cert/apiserver.crt, /var/run/serving-cert/apiserver.key)
I0506 11:28:30.131352       1 serve.go:96] Serving securely on [::]:6443
E0506 11:28:53.628657       1 writers.go:149] apiserver was unable to write a JSON response: http2: stream closed
E0506 11:28:53.628687       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0506 11:28:53.628699       1 writers.go:149] apiserver was unable to write a JSON response: http2: stream closed
E0506 11:28:53.630557       1 writers.go:149] apiserver was unable to write a JSON response: http2: stream closed
E0506 11:28:53.630851       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0506 11:28:53.632975       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

Did you expect to see some different?
No errors
How to reproduce it (as minimally and precisely as possible):
kubectl apply -f manifests
Environment

Prometheus Adapter version:
v0.5.0 , Same issue with v0.6.0,
adapter log similar error and then panics with v0.7.0 , error logs are

I0507 04:28:55.121299       1 dynamic_serving_content.go:129] Starting serving-cert::/var/run/serving-cert/apiserver.crt::/var/run/serving-cert/apiserver.key
I0507 04:28:55.121318       1 tlsconfig.go:219] Starting DynamicServingCertificateController
I0507 04:28:55.121548       1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0507 04:28:55.121558       1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0507 04:28:55.221420       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0507 04:28:55.221677       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E0507 04:29:23.979358       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0507 04:29:23.979384       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0507 04:29:23.980432       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0507 04:29:23.980764       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 356 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0033c0180)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/server/filters/timeout.go:108 +0x107
panic(0x1861bc0, 0x2a840f0)
	/usr/lib/golang/src/runtime/panic.go:679 +0x1b2
compress/gzip.(*Writer).Write(0xc003226000, 0xc003388f80, 0x71, 0x71, 0x30, 0x1920b00, 0xc0037a8d01)
	/usr/lib/golang/src/compress/gzip/gzip.go:168 +0x237
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.(*deferredResponseWriter).Write(0xc001ebc0a0, 0xc003388f80, 0x71, 0x71, 0xc003388f80, 0x71, 0x71)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/handlers/responsewriters/writers.go:182 +0x54e
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1ab3a63, 0x10, 0x7fa9dc4292c8, 0xc002b481e0, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300, 0xc8, 0x1d597a0, 0xc001ebc050)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/handlers/responsewriters/writers.go:117 +0x389
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x1d8e6e0, 0xc0005d50e0, 0x1d8e920, 0x2ac4d18, 0x0, 0x0, 0x0, 0x0, 0x7fa9dc6a8ed8, 0xc0001fc020, ...)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/handlers/responsewriters/writers.go:251 +0x555
k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).ServeHTTP(0xc003a22d80, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/discovery/version.go:81 +0x18f
k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).handle(...)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/discovery/version.go:77
github.com/emicklei/go-restful.(*Container).dispatch(0xc0005579e0, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/home/sur/src/redhat/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:288 +0xa4f
github.com/emicklei/go-restful.(*Container).Dispatch(...)
	/home/sur/src/redhat/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:199
k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1ac198d, 0x1a, 0xc0005579e0, 0xc0004bebd0, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/server/handler.go:146 +0x4d3
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/filters/authorization.go:64 +0x512
net/http.HandlerFunc.ServeHTTP(0xc000491dc0, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/usr/lib/golang/src/net/http/server.go:2007 +0x44
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/server/filters/maxinflight.go:160 +0x5dc
net/http.HandlerFunc.ServeHTTP(0xc001cdc750, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/usr/lib/golang/src/net/http/server.go:2007 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/filters/impersonation.go:50 +0x1fe6
net/http.HandlerFunc.ServeHTTP(0xc000491e00, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646300)
	/usr/lib/golang/src/net/http/server.go:2007 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646200)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/filters/authentication.go:131 +0xa8c
net/http.HandlerFunc.ServeHTTP(0xc0004cf4a0, 0x7fa9dc6a8ed8, 0xc0001fc020, 0xc003646200)
	/usr/lib/golang/src/net/http/server.go:2007 +0x44
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0033c0180, 0xc001cd69c0, 0x1d93160, 0xc0001fc020, 0xc003646200)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/server/filters/timeout.go:113 +0xd0
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/server/filters/timeout.go:99 +0x1cb

Prometheus Operator version:
v0.38.1

Kubernetes version information:
v1.17.3

Similar issue found
https://github.com/helm/charts/issues/18576

encountered the same issue (apiserver was unable to write a JSON response) on k8s 1.15.
i noticed that the chart creates a RoleBinding to extension-apiserver-authentication-reader
i have this role in namespace kube-system not in the namespace where i create the adapter (prometheus).
i tried creating the roleBinding in kube-system namepace - but it did not help.
@DirectXMan12 : do you think this error is authorization based (and i just did not fix it well enough) or is it something else (e.g. dns resolution)?

Strange. We're running the latest version as well with Kubernetes 1.18.2, but not seeing this or the panic. Could you possibly try with that version and see if you still get the same errors and panics?

@brancz : if it is not too much, coule you please share if the prometheus-adapter is deployed in kube-system namespace or elsewhere?
if elsewhere - how did prometheus-adapter-auth-reader RoleBinding work if at all?

Here's our RoleBinding: https://github.com/coreos/kube-prometheus/blob/3c109369d4d4e19d2bab9e073797eb971a5eb957/manifests/prometheus-adapter-roleBindingAuthReader.yaml

the roleBinding created by helm happens is in the adapter namespace (monitoring in your case) not kube-system.
Changed my binding to the one you have – and restarted pod.
Still getting the “apiserver was unable to write a JSON response: http2: stream closed” messages.
Maybe the issue is not permissions…
If not permissions, it must be networking, right?
Regards,
Daniel

@brancz @daniel-habib Sorry for the late response.
I ran some tests using kube-prometheus:

if kube-promehteus stack is deployed in the default namespace which is monitoring, even with custom metrics enabled(import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet'), there is no panic and apiserver was unable to write a JSON response: http2: stream closed error.
if kube-promehteus stack is deployed in a different namespace by changing examples/kustomize.jsonnet without enabling custom metrics, everything is fine. No panic and no error logs
if kube-promehteus stack is deployed in a different namespace with custom metrics enabled(import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet'), then there is panic and error messages like below. And if I change the prometheus adapter image from v0.7.0 to v0.6.0, then no panic anymore, only  error log like apiserver was unable to write a JSON response: http2: stream closed exists

So for me it seems that this is related to prometheus adapter custom metrics functionality and namespace
@daniel-habib did you deploy prometheus adapter in a different namespace instead of the default monitoring namespace?
I0601 08:03:05.113483       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0601 08:03:05.113483       1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E0601 08:03:23.519665       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.519722       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.521181       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E0601 08:03:23.526186       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.527525       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.527708       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.528301       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.528639       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.528732       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E0601 08:03:23.528970       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.529429       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.530499       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.530852       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.531980       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.533048       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.533268       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E0601 08:03:23.534207       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.535325       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.536514       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0601 08:03:23.537787       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 698 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0003a3b00)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/server/filters/timeout.go:108 +0x107
panic(0x1861bc0, 0x2a840f0)
	/usr/lib/golang/src/runtime/panic.go:679 +0x1b2
compress/gzip.(*Writer).Write(0xc000218dc0, 0xc003384100, 0x71, 0x71, 0x30, 0x1920b00, 0xc0009bad01)
	/usr/lib/golang/src/compress/gzip/gzip.go:168 +0x237
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.(*deferredResponseWriter).Write(0xc00043c3c0, 0xc003384100, 0x71, 0x71, 0xc003384100, 0x71, 0x71)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/handlers/responsewriters/writers.go:182 +0x54e
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1ab3a63, 0x10, 0x7f32b6596520, 0xc000471400, 0x7f32b6596490, 0xc0004b6e20, 0xc0002b0100, 0xc8, 0x1d597a0, 0xc00043c2d0)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/handlers/responsewriters/writers.go:117 +0x389
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x1d8e6e0, 0xc00054ecc0, 0x1d8e920, 0x2ac4d18, 0x0, 0x0, 0x0, 0x0, 0x7f32b6596490, 0xc0004b6e20, ...)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/handlers/responsewriters/writers.go:251 +0x555
k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).ServeHTTP(0xc0002ee9c0, 0x7f32b6596490, 0xc0004b6e20, 0xc0002b0100)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/discovery/version.go:81 +0x18f
k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).handle(...)
	/home/sur/src/redhat/go/pkg/mod/k8s.io/apiserver@v0.17.3/pkg/endpoints/discovery/version.go:77
github.com/emicklei/go-restful.(*Container).dispatch(0xc00067d4d0, 0x7f32b6596490, 0xc0004b6e20, 0xc0002b0100)
	/home/sur/src/redhat/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:288 +0xa4f
github.com/emicklei/go-restful.(*Container).Dispatch(...)
	/home/sur/src/redhat/go/pkg/mod/github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:199
k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1ac198d, 0x1a, 0xc00067d4d0, 0xc0004cc690, 0x7f32b6596490, 0xc0004b6e20, 0xc0002b0100)

@benjaminhuo : i deployed it via helm and it was deployed in namespace "prometheus"
i'll try to see if dropping custom metrics fixes the issue - and report back later.

the helm chart created this config.yaml
rules:
- seriesQuery: '{__name__=~"^container_.*",container!="POD",namespace!="",pod!=""}'
  seriesFilters: []
  resources:
    overrides:
      namespace:
        resource: namespace
      pod:
        resource: pod
  name:
    matches: ^container_(.*)_seconds_total$
    as: ""
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[5m]))
    by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container!="POD",namespace!="",pod!=""}'
  seriesFilters:
  - isNot: ^container_.*_seconds_total$
  resources:
    overrides:
      namespace:
        resource: namespace
      pod:
        resource: pod
  name:
    matches: ^container_(.*)_total$
    as: ""
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[5m]))
    by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container!="POD",namespace!="",pod!=""}'
  seriesFilters:
  - isNot: ^container_.*_total$
  resources:
    overrides:
      namespace:
        resource: namespace
      pod:
        resource: pod
  name:
    matches: ^container_(.*)$
    as: ""
  metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container!="POD"}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
  seriesFilters:
  - isNot: .*_total$
  resources:
    template: <<.Resource>>
  name:
    matches: ""
    as: ""
  metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
  seriesFilters:
  - isNot: .*_seconds_total
  resources:
    template: <<.Resource>>
  name:
    matches: ^(.*)_total$
    as: ""
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
  seriesFilters: []
  resources:
    template: <<.Resource>>
  name:
    matches: ^(.*)_seconds_total$
    as: ""
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)

when i switched to
rules:
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
  seriesFilters: []
  resources:
    template: <<.Resource>>
  name:
    matches: ^(.*)_seconds_total$
    as: ""
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)

errors disappeared.
perhaps the helm's package default is too heavy (as suggested in https://github.com/DirectXMan12/k8s-prometheus-adapter/issues/264#issuecomment-596584155)?

@daniel-habib yeah, I confirmed your findings as well. So @brancz @DirectXMan12 , it seems that this error is related to the data volume of metricsQuery

Yeah we're aware of problems with the scalability of the current approach, because of the discovery mechanism. I believe @s-urbaniak wrote a proposal to replace this mechanism, maybe it can be shared publicly so anyone who wants to could implement it?

That'll be great. I'll see how can I help on this.

@benjaminhuo help here is very much appreciated, the initial design is available at https://github.com/s-urbaniak/prometheus-adapter/blob/master/design.md.
Don't hesitate to take it from here, I'll gladly accept pull request against the design doc and any subsequent discussions :raised_hands: :heart:

@s-urbaniak Sure, let me check the design

@benjaminhuo also, don't hesitate to ask questions, clarifications, i.e. by opening an issue against my repo https://github.com/s-urbaniak/prometheus-adapter.

@benjaminhuo also, don't hesitate to ask questions, clarifications, i.e. by opening an issue against my repo https://github.com/s-urbaniak/prometheus-adapter.

Got that

@benjaminhuo @daniel-habib  i had the same error as Prometheus Adapter chart was creating default rules. You can avoid that by using this chart configuration :
# Default value is true
rules.default=false

Here is my configuration :
prometheus:
  url: http://prometheus-server.prometheus.svc
  port: 80

image:
  tag: v0.7.0

rbac:
  create: true

logLevel: 10

rules:

# This one is important and will remove the default rules.
  default: false
  external:
  - seriesQuery: '{__name__=~"^buildkite_queues_scheduled_jobs_count$"}'
    resources:
      overrides:
        job: {resource: "namespace"}
        queue: {resource: "pod"}
    name:
      matches: ""
      as: "buildkite_queues_scheduled_jobs_count"
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)

With this only configuration that gets created is :
cat /etc/adapter/config.yaml

externalRules:
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  name:
    as: buildkite_queues_scheduled_jobs_count
    matches: ""
  resources:
    overrides:
      job:
        resource: namespace
      queue:
        resource: pod
  seriesQuery: '{__name__=~"^buildkite_queues_scheduled_jobs_count$"}'

@benjaminhuo @daniel-habib i had the same error as Prometheus Adapter chart was creating default rules. You can avoid that by using this chart configuration :
(...)

I'm running k8s 1.18.3 and was brought here by the same issue.
Your fix @shardulsrivastava works wonders.
Thank you!

/cc @DirectXMan12 @brancz I have the exact same problem (1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference goroutine 698 [running]:),while  trying to deploy prometheus-adapter in kops. I used the manifests from kube-prometheus. Tried to deploy it in prometheus and kube-system namespaces with no success. Previously had the same setup on EKS and worked as expected. A main difference is that on eks used monitoring namespace. Any ideas on how i could debug this?

cc @s-urbaniak

hmm, i have never observed this behavior, let me dig into the code, maybe i'll spot something :thinking:

@stafot can you please also submit the whole stack trace? is it the same as above?

while doing some investigation together with @p0lyn0mial we figure the following fix might fix the issue here as well: https://github.com/kubernetes/kubernetes/pull/94589

The above fix will be included in k8s 1.20 so we can re-asses once we bump to that version which should happen shortly.

@s-urbaniak All the log file, including traces.
I1209 14:43:50.281836       1 adapter.go:98] successfully using in-cluster auth
I1209 14:43:51.927069       1 serving.go:325] Generated self-signed cert (/var/run/serving-cert/apiserver.crt, /var/run/serving-cert/apiserver.key)
I1209 14:43:53.505650       1 secure_serving.go:197] Serving securely on [::]:6443
I1209 14:43:53.506125       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1209 14:43:53.506141       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1209 14:43:53.506174       1 dynamic_serving_content.go:130] Starting serving-cert::/var/run/serving-cert/apiserver.crt::/var/run/serving-cert/apiserver.key
I1209 14:43:53.506194       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1209 14:43:53.510185       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1209 14:43:53.510203       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1209 14:43:53.510220       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1209 14:43:53.510224       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1209 14:43:53.683832       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1209 14:43:53.684049       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I1209 14:43:53.684252       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E1209 14:43:55.008081       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:43:55.008263       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:43:55.008795       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:43:55.019139       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:43:55.039474       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:43:55.041591       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:43:55.042656       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:43:55.043752       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:43:55.044817       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:44:14.756519       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:44:14.756618       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:44:14.757152       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:44:14.757565       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:44:14.758249       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:44:14.758477       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:44:14.760411       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:44:14.761483       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:44:14.762575       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:44:14.763664       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:44:14.764714       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:44:14.764803       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:44:14.765866       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:44:14.768007       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:44:14.771250       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:44:44.756251       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:44:44.756282       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:44:44.759109       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:44:44.768570       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:44:44.768594       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:44:44.769682       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:45:14.757030       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:45:14.757177       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:45:14.757908       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:45:14.761947       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:45:14.764103       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:45:14.765196       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:45:44.763959       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:45:44.764167       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:45:44.766049       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:46:14.758106       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:46:14.758140       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:46:14.758672       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:46:14.759038       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:46:14.759548       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:46:14.759624       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:46:14.761952       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:46:14.763040       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:46:14.764125       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:46:14.765334       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:46:14.767321       1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1209 14:46:14.767480       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:46:14.768558       1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1209 14:46:14.769637       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1209 14:46:14.773087       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 1813 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0016aafc0)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x107
panic(0x1aa1b60, 0x3189600)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
compress/gzip.(*Writer).Write(0xc0020c9340, 0xc002af4f80, 0x71, 0x71, 0x30, 0x1b79a80, 0xc0021fcc01)
	/usr/local/go/src/compress/gzip/gzip.go:168 +0x237
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.(*deferredResponseWriter).Write(0xc00177b810, 0xc002af4f80, 0x71, 0x71, 0xc002af4f80, 0x71, 0x71)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:182 +0x54e
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1d35b02, 0x10, 0x7fb73e377710, 0xc0004ef180, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00, 0xc8, 0x2096640, 0xc00177b770)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:117 +0x389
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x20b9640, 0xc0005d1b00, 0x20b98c0, 0x31ce040, 0x0, 0x0, 0x0, 0x0, 0x7fb73e373530, 0xc00000ed90, ...)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:251 +0x555
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).ServeHTTP(0xc0007db340, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery/version.go:81 +0x17b
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).handle(...)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery/version.go:77
github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc000107290, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful/container.go:294 +0x635
github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(...)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful/container.go:204
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1d44a83, 0x1a, 0xc000107290, 0xc000124850, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x531
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x548
net/http.HandlerFunc.ServeHTTP(0xc003187e80, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func2(0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:175 +0x4f6
net/http.HandlerFunc.ServeHTTP(0xc000d90930, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x22f3
net/http.HandlerFunc.ServeHTTP(0xc003187ec0, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fb73e373530, 0xc00000ed90, 0xc00197ed00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x5ea
net/http.HandlerFunc.ServeHTTP(0xc00017d9f0, 0x7fb73e373530, 0xc00000ed90, 0xc00197ed00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0016aafc0, 0xc0023c1720, 0x20c3800, 0xc00000ed90, 0xc00197ed00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:113 +0xd0
created by github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:99 +0x1cb

goroutine 1781 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19fddc0, 0xc001765050)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc000495c90, 0x1, 0x1)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x19fddc0, 0xc001765050)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc0023c1720, 0x20b7540, 0xc000269180, 0xc00197ed00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:119 +0x486
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x20b7540, 0xc000269180, 0xc00197ec00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x121
net/http.HandlerFunc.ServeHTTP(0xc000d90960, 0x20b7540, 0xc000269180, 0xc00197ec00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x20b7540, 0xc000269180, 0xc00197ea00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x274
net/http.HandlerFunc.ServeHTTP(0xc000d90990, 0x20b7540, 0xc000269180, 0xc00197ea00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x20b7540, 0xc000269180, 0xc00197e900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a0
net/http.HandlerFunc.ServeHTTP(0xc0023c1740, 0x20b7540, 0xc000269180, 0xc00197e900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x20b7540, 0xc000269180, 0xc00197e900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc0023c1760, 0x20b7540, 0xc000269180, 0xc00197e900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x20aac40, 0xc0016869a8, 0xc000821900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2a4
net/http.HandlerFunc.ServeHTTP(0xc0023c1780, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x20aac40, 0xc0016869a8, 0xc000821900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0x13e
net/http.HandlerFunc.ServeHTTP(0xc0023c17a0, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc000d909c0, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc002bf7b20, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:2831 +0xa4
net/http.initNPNRequest.ServeHTTP(0x20c3100, 0xc001b941e0, 0xc001b96e00, 0xc002bf7b20, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:3395 +0x8d
github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc001db6000, 0xc0016869a8, 0xc000821900, 0xc0016dbb80)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:2147 +0x9f
created by github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:1881 +0x4e5
E1209 14:46:14.773115       1 wrap.go:39] apiserver panic'd on GET /apis/custom.metrics.k8s.io/v1beta1
http2: panic serving <IP_ADDRESS>:55214: runtime error: invalid memory address or nil pointer dereference
goroutine 1813 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0016aafc0)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x107
panic(0x1aa1b60, 0x3189600)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
compress/gzip.(*Writer).Write(0xc0020c9340, 0xc002af4f80, 0x71, 0x71, 0x30, 0x1b79a80, 0xc0021fcc01)
	/usr/local/go/src/compress/gzip/gzip.go:168 +0x237
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.(*deferredResponseWriter).Write(0xc00177b810, 0xc002af4f80, 0x71, 0x71, 0xc002af4f80, 0x71, 0x71)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:182 +0x54e
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1d35b02, 0x10, 0x7fb73e377710, 0xc0004ef180, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00, 0xc8, 0x2096640, 0xc00177b770)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:117 +0x389
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x20b9640, 0xc0005d1b00, 0x20b98c0, 0x31ce040, 0x0, 0x0, 0x0, 0x0, 0x7fb73e373530, 0xc00000ed90, ...)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:251 +0x555
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).ServeHTTP(0xc0007db340, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery/version.go:81 +0x17b
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery.(*APIVersionHandler).handle(...)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/discovery/version.go:77
github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc000107290, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful/container.go:294 +0x635
github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(...)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful/container.go:204
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1d44a83, 0x1a, 0xc000107290, 0xc000124850, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x531
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x548
net/http.HandlerFunc.ServeHTTP(0xc003187e80, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func2(0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:175 +0x4f6
net/http.HandlerFunc.ServeHTTP(0xc000d90930, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x22f3
net/http.HandlerFunc.ServeHTTP(0xc003187ec0, 0x7fb73e373530, 0xc00000ed90, 0xc00197ee00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fb73e373530, 0xc00000ed90, 0xc00197ed00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x5ea
net/http.HandlerFunc.ServeHTTP(0xc00017d9f0, 0x7fb73e373530, 0xc00000ed90, 0xc00197ed00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0016aafc0, 0xc0023c1720, 0x20c3800, 0xc00000ed90, 0xc00197ed00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:113 +0xd0
created by github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:99 +0x1cb

goroutine 1781 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2.(*serverConn).runHandler.func1(0xc0016869a8, 0xc000495f67, 0xc001db6000)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:2140 +0x16b
panic(0x19fddc0, 0xc001765050)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc000495c90, 0x1, 0x1)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x19fddc0, 0xc001765050)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc0023c1720, 0x20b7540, 0xc000269180, 0xc00197ed00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:119 +0x486
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x20b7540, 0xc000269180, 0xc00197ec00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x121
net/http.HandlerFunc.ServeHTTP(0xc000d90960, 0x20b7540, 0xc000269180, 0xc00197ec00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x20b7540, 0xc000269180, 0xc00197ea00)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x274
net/http.HandlerFunc.ServeHTTP(0xc000d90990, 0x20b7540, 0xc000269180, 0xc00197ea00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x20b7540, 0xc000269180, 0xc00197e900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a0
net/http.HandlerFunc.ServeHTTP(0xc0023c1740, 0x20b7540, 0xc000269180, 0xc00197e900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x20b7540, 0xc000269180, 0xc00197e900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc0023c1760, 0x20b7540, 0xc000269180, 0xc00197e900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x20aac40, 0xc0016869a8, 0xc000821900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2a4
net/http.HandlerFunc.ServeHTTP(0xc0023c1780, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x20aac40, 0xc0016869a8, 0xc000821900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0x13e
net/http.HandlerFunc.ServeHTTP(0xc0023c17a0, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:2036 +0x44
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc000d909c0, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc002bf7b20, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:2831 +0xa4
net/http.initNPNRequest.ServeHTTP(0x20c3100, 0xc001b941e0, 0xc001b96e00, 0xc002bf7b20, 0x20aac40, 0xc0016869a8, 0xc000821900)
	/usr/local/go/src/net/http/server.go:3395 +0x8d
github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc001db6000, 0xc0016869a8, 0xc000821900, 0xc0016dbb80)
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:2147 +0x9f
created by github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:1881 +0x4e5

@stafot thank you! yes, it seems that this is the same issue then and should be fixed (famous last words) in the next bump of the api machinery dependencies.

@s-urbaniak If I remember correctly, this should have been fixed by #352. Considering that a lot of users are impacted by this bug, it might be worth cutting v0.8.3 to include the fix.

@s-urbaniak If I remember correctly, this should have been fixed by #352. Considering that a lot of users are impacted by this bug, it might be worth cutting v0.8.3 to include the fix.

I use the latest code in the k8s v1.17.2, it still have error.
E0126 07:09:33.098534       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0126 07:09:33.099247       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0126 07:09:33.099588       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0126 07:09:33.100127       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0126 07:09:33.100601       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0126 07:09:33.101705       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
I0126 07:09:33.102881       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="21.862952ms" userAgent="Go-http-client/2.0" srcIP="<IP_ADDRESS>:50383" resp=200
E0126 07:09:33.103895       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0126 07:09:33.104972       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0126 07:09:33.106066       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0126 07:09:33.107160       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
I0126 07:09:33.108416       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="27.468943ms" userAgent="Go-http-client/2.0" srcIP="<IP_ADDRESS>:50383" resp=200

I use the latest code in the k8s v1.17.2, it still have error.
E0126 07:09:33.098534       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0126 07:09:33.099247       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0126 07:09:33.099588       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0126 07:09:33.100127       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0126 07:09:33.100601       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0126 07:09:33.101705       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
I0126 07:09:33.102881       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="21.862952ms" userAgent="Go-http-client/2.0" srcIP="<IP_ADDRESS>:50383" resp=200
E0126 07:09:33.103895       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0126 07:09:33.104972       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0126 07:09:33.106066       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0126 07:09:33.107160       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
I0126 07:09:33.108416       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="27.468943ms" userAgent="Go-http-client/2.0" srcIP="<IP_ADDRESS>:50383" resp=200

@dgrisonnet agreed, and because we have #362 as well I will cut a new release today :+1:

@dgrisonnet agreed, and because we have #362 as well I will cut a new release today :+1:

@pytimer yes the error is still present. The fix that we were able to find basically only "silent" the panics, but the symptom is still here. So far, it doesn't seem to have an actual impact on how prometheus-adapter run, so it might be something very normal such as a client closing the connection.
Also, it might be worth noting that this error was seen elsewhere in the ecosystem consuming custom-metrics-apiserver, so the symptom is most likely coming from there.

@pytimer yes the error is still present. The fix that we were able to find basically only "silent" the panics, but the symptom is still here. So far, it doesn't seem to have an actual impact on how prometheus-adapter run, so it might be something very normal such as a client closing the connection.
Also, it might be worth noting that this error was seen elsewhere in the ecosystem consuming custom-metrics-apiserver, so the symptom is most likely coming from there.

Also, it might be worth noting that this error was seen elsewhere in the ecosystem consuming custom-metrics-apiserver, so the symptom is most likely coming from there.

I cannot found relevant informantion in the adapter, so i think maybe it related to custom-metrics-apisever.
Use the latest code, panic go away.

Also, it might be worth noting that this error was seen elsewhere in the ecosystem consuming custom-metrics-apiserver, so the symptom is most likely coming from there.

I cannot found relevant informantion in the adapter, so i think maybe it related to custom-metrics-apisever.
Use the latest code, panic go away.

Note: we just released v0.8.3 so please try it out if it solves the issue, closing out for the time being :-)

Note: we just released v0.8.3 so please try it out if it solves the issue, closing out for the time being :-)

Running v0.8.3 and I bumped into this issue too
E0201 21:20:16.900344       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0201 21:20:16.902991       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0201 21:20:16.905160       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0201 21:20:16.905531       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0201 21:20:16.907333       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0201 21:20:16.908808       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0201 21:20:46.912815       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0201 21:20:46.912848       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0201 21:20:46.913995       1 writers.go:101] apiserver was unable to close cleanly the response writer: http2: stream closed
E0201 21:20:46.914030       1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed
E0201 21:20:46.914354       1 writers.go:101] apiserver was unable to close cleanly the response writer: http2: stream closed
E0201 21:20:46.915218       1 writers.go:101] apiserver was unable to close cleanly the response writer: http2: stream closed
E0201 21:20:46.916360       1 writers.go:101] apiserver was unable to close cleanly the response writer: http2: stream closed
E0201 21:20:46.916582       1 writers.go:101] apiserver was unable to close cleanly the response writer: http2: stream closed

Replicated the error with nginx proxy_pass with extra (upgrade) headers. Fixed it with a custom location /apis/custom.metrics.k8s.io/ w/o the headers.

@s-urbaniak Can you re-open this issue? It's still in v0.8.3.

@s-urbaniak We also have the same issue in v0.8.3 on K8S 1.19, would be nice to reopen

I am seeing this issue in prometheus adapter v0.8.4 with Kubernetes API server 1.20.

At pod restart, there are no errors and pod CPU util is fine, but after a few minutes errors start again and pod starts routinely CPU spiking.

These spikes are entirely attributed to the adapter. Is anyone else seeing this behavior?

I see the same issue as well in v0.8.4 and kubernetes 1.20 in eks with another namespace
Could you reopen this?

I can confirm this issue is still in 0.9.0. I'm running on kube 1.20.10.

Same here with prometheus-community helm chart 2.17.0 and prometheus adapter 0.9.0 on k8s 1.17.
@shardulsrivastava 's workaround is a charm but you of course give up on those additional rules.

I'm also facing this problem with prometheus-community helm chart 2.17.0, prometheus-adapter 0.9.0 and k8s v1.21.2-eks-0389ca3 (it is AWS EKS 1.21)
Only solution is @shardulsrivastava 's workaround, but this drop major prometheus-adapter functionality, which we need. :(

Still facing the same bugs. Any help? Please re-open.

Running v0.9.0 on k8s 1.17 and am seeing the same error. Adapter starts fine but it then immediately logs this:
I1011 22:08:06.557982       1 adapter.go:108] successfully using in-cluster auth
I1011 22:08:07.250481       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1011 22:08:07.250502       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1011 22:08:07.250497       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1011 22:08:07.250525       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1011 22:08:07.250550       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1011 22:08:07.250559       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1011 22:08:07.250886       1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::/var/run/serving-cert/serving.crt::/var/run/serving-cert/serving.key"
I1011 22:08:07.251175       1 secure_serving.go:256] Serving securely on [::]:6443
I1011 22:08:07.251194       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1011 22:08:07.350692       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1011 22:08:07.350713       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1011 22:08:07.350731       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
E1011 22:08:09.520231       1 writers.go:117] apiserver was unable to write a JSON response: http2: stream closed
E1011 22:08:09.520259       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E1011 22:08:09.520235       1 writers.go:117] apiserver was unable to write a JSON response: http2: stream closed
E1011 22:08:09.520267       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="106288bc-c569-476b-87ce-a2c87b3802b9"
E1011 22:08:09.520292       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="330cd927-1ab7-4d6f-9a40-8e9151468911"

at regular intervals

Another vote from me that this is still an easy with k8s 1.20

This issue was closed because the original panic that was reported was fixed in v0.8.3. Since the symptoms that you are reporting are similar to the original issue but the problem in itself is different, I would prefer that you file a separate issue since this one has been reported different problems over time.
Also, as I mentioned before in https://github.com/kubernetes-sigs/prometheus-adapter/issues/292#issuecomment-767444375, we know that the symptoms are still present but they don't seem to have any impact on prometheus-adapter besides spamming the logs.

I'm also facing the same issues. is there any solution, please.
prometheus-adapter:v0.9.1 and Kubernetes v1.21.5
error :
apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

The problem still exist - EKS 1.22 + prometheus-adapter:v0.9.1

We are experiencing this error as well in Eks 1.21 +  prometheus-adapter:v0.9.1. Also, we have to allocate huge amount of memory (4G in our Production env) for each prometheus_adapter pod to avoid OOM. Not sure if this error has anything to do with that.

same stuff here, DigitalOcean managed k8s 1.23, prometheus adapter version v0.10.0

The original issue mentioned a panic which was fixed by https://github.com/kubernetes/kubernetes/pull/94589 and then vendored in https://github.com/kubernetes-sigs/prometheus-adapter/pull/352.
We investigated the spammy logs in the past but haven't been able to exactly pinpoint where the issue is coming from. I summed up the status of the investigation in https://github.com/kubernetes-sigs/prometheus-adapter/issues/292#issuecomment-767444375.

Are you sure that the fix you mentionned is sufficient ?

The panic doesn't happen anymore so yes it is. That said the logs are still present but so far I haven't seen any disruption coming from them nor I have been reported any. So I am pretty confident to say that it is pretty harmless.

As I mentioned multiple times in the past, this issue will remain close as the original issue (panic) was fixed. However if anyone is interested in having the spammy logs being investigated feel free to open another one. But, I don't think we have the resources to investigate it today.

If you are seeing this in your logs and your pods are in a CrashLoopBackOff then you probably need to increase the amount of memory the prometheus-adapter has allocated. Out of the box the helm chart does not specify enough to handle the default ruleset it ships with.

Still happening on version 0.10.0 with K8s api-server 1.23.5

Still happening

Still happening on version 0.10.0 with Azure AKS 1.24.6 Kubernetes version.

Still happening on v0.10.0 (Helm chart v4.2.0) w/ kubeadm-managed cluster v1.24.15
The solution proposed in here does remove the error from logs, but kubectl top no command still not work with error: Metrics API not available error
any update?

Same issue for me

If you are seeing this in your logs and your prometheus-adapter pod is in a CrashLoopBackOff then you probably need to increase the amount of memory that prometheus-adapter has allocated. Out of the box the helm chart does not specify enough to handle the default ruleset it ships with.

I have confirmed that this does not fix the problem.
This is not a resource constraint issue

Same issue on Kube 1.26.11

Any news on this? I'm currently on rancher 2.7.9 with rke2 v1.26.12 and the current helm chart rancher-monitoring:102.0.2+up40.1.2 (which uses rancher/mirrored-prometheus-adapter-prometheus-adapter:v0.10.0) the following logs were still created:
0125 12:09:41.519596       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0125 12:09:41.521726       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0125 12:09:41.523854       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0125 12:09:41.526064       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout

Are there any debug-options available for the pod to show the URL/api-call that causes this?

I am experiencing the same issue with rancher-monitoring,
Rancher version: 2.7.9
K8S version: v1.26.10+k3s1
Rancher monitoring version: 102.0.2+up40.1.2
OS: Ubuntu 22.04.3 LTS
Hardware: Raspberry Pi (8 nodes, 3 cp/etcd/master nodes)
Arch: AArch64/Arm64
Logs:
 E0129 00:40:22.839747       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:40:22.840815       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
2024-01-29T01:40:22.843431070+01:00 E0129 00:40:22.843301       1 timeout.go:141] post-timeout activity - time-elapsed: 16.912179ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:22.845581483+01:00 E0129 00:40:22.845462       1 timeout.go:141] post-timeout activity - time-elapsed: 18.091347ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:40:22.846400       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:40:22.850180       1 timeout.go:141] post-timeout activity - time-elapsed: 16.516334ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
I0129 00:40:24.720804       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="61.441976ms" userAgent="k3s/v1.26.10+k3s1 (linux/arm64) kubernetes/a6687b5/system:serviceaccount:kube-system:generic-garbage-collector" audit-ID="a0353424-f579-45a4-860b-769e5279db7f" srcIP="<IP_ADDRESS>:52170" resp=200
2024-01-29T01:40:25.058699940+01:00 I0129 00:40:25.058478       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="949.19µs" userAgent="kube-probe/1.26" audit-ID="b953a203-32a0-4882-b49a-654dbcfc3821" srcIP="<IP_ADDRESS>:36336" resp=200
I0129 00:40:30.151406       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="6.596923ms" userAgent="kube-probe/1.26" audit-ID="6a478af0-2ed6-4540-a4b1-a8823e36d255" srcIP="<IP_ADDRESS>:40240" resp=200
2024-01-29T01:40:33.229844405+01:00 I0129 00:40:33.229586       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="61.736026ms" userAgent="k3s/v1.26.10+k3s1 (linux/arm64) kubernetes/a6687b5/controller-discovery" audit-ID="ed7075a3-6fa2-444b-a04d-43042ae3d655" srcIP="<IP_ADDRESS>:52170" resp=200
I0129 00:40:33.533645       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="47.088074ms" userAgent="Go-http-client/2.0" audit-ID="2c419a8c-e316-4f3b-8ab6-87cf7b32eb01" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:40:33.534845055+01:00 E0129 00:40:33.534580       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="2c419a8c-e316-4f3b-8ab6-87cf7b32eb01"
2024-01-29T01:40:33.535478398+01:00 E0129 00:40:33.535238       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
2024-01-29T01:40:33.536140481+01:00 E0129 00:40:33.535757       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
I0129 00:40:33.535909       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="49.612703ms" userAgent="Go-http-client/2.0" audit-ID="6e517dfd-31ca-42aa-b770-6e5d1d118ba7" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:40:33.536400514+01:00 E0129 00:40:33.536244       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="6e517dfd-31ca-42aa-b770-6e5d1d118ba7"
2024-01-29T01:40:33.536541197+01:00 E0129 00:40:33.535961       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
2024-01-29T01:40:33.536853433+01:00 E0129 00:40:33.536647       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
I0129 00:40:33.536707       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="39.204892ms" userAgent="Go-http-client/2.0" audit-ID="963ac0a9-2b13-42a8-9eaa-502d9f13f91d" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:40:33.537300482+01:00 E0129 00:40:33.537051       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:40:33.537071       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="963ac0a9-2b13-42a8-9eaa-502d9f13f91d"
2024-01-29T01:40:33.539514302+01:00 E0129 00:40:33.539256       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
2024-01-29T01:40:33.541789584+01:00 E0129 00:40:33.541464       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
2024-01-29T01:40:33.546769696+01:00 E0129 00:40:33.546366       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
I0129 00:40:33.547478       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="61.135554ms" userAgent="Go-http-client/2.0" audit-ID="7b31132b-bd5c-4d57-9e88-13e9f44a0c73" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:40:33.548481671+01:00 E0129 00:40:33.547497       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
2024-01-29T01:40:33.548599799+01:00 E0129 00:40:33.548249       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="7b31132b-bd5c-4d57-9e88-13e9f44a0c73"
E0129 00:40:33.551166       1 timeout.go:141] post-timeout activity - time-elapsed: 15.647513ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:40:33.552148       1 timeout.go:141] post-timeout activity - time-elapsed: 15.513755ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:40:33.553164       1 timeout.go:141] post-timeout activity - time-elapsed: 16.321206ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:33.553487654+01:00 E0129 00:40:33.553332       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
2024-01-29T01:40:33.554498510+01:00 I0129 00:40:33.554293       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="68.445189ms" userAgent="Go-http-client/2.0" audit-ID="5c4a392b-c1ec-46a4-8cee-3a3749a7dc4d" srcIP="<IP_ADDRESS>:36818" resp=200
E0129 00:40:33.554365       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="5c4a392b-c1ec-46a4-8cee-3a3749a7dc4d"
2024-01-29T01:40:33.554601045+01:00 E0129 00:40:33.554442       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0129 00:40:33.558481       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:40:33.560710       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:40:33.561810       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
2024-01-29T01:40:33.563051422+01:00 E0129 00:40:33.562956       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:40:33.564417       1 timeout.go:141] post-timeout activity - time-elapsed: 10.020557ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:40:33.565538       1 timeout.go:141] post-timeout activity - time-elapsed: 17.982683ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:35.060179136+01:00 I0129 00:40:35.059964       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="2.983938ms" userAgent="kube-probe/1.26" audit-ID="362bc583-d729-4b0b-945b-066fd36662b1" srcIP="<IP_ADDRESS>:40254" resp=200
I0129 00:40:38.474642       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="59.369987ms" userAgent="k3s/v1.26.10+k3s1 (linux/arm64) kubernetes/a6687b5/system:serviceaccount:kube-system:resourcequota-controller" audit-ID="bc3f54f2-ee14-4eb8-ae19-18e239398b0f" srcIP="<IP_ADDRESS>:52170" resp=200
I0129 00:40:40.148542       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="3.248694ms" userAgent="kube-probe/1.26" audit-ID="9f19bd55-8ab3-4d6d-ac60-4ad23e3a932b" srcIP="<IP_ADDRESS>:47448" resp=200
2024-01-29T01:40:45.065923875+01:00 I0129 00:40:45.065688       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="10.236832ms" userAgent="kube-probe/1.26" audit-ID="7d5f2e02-4da0-4549-9b50-989d2186eaf4" srcIP="<IP_ADDRESS>:47460" resp=200
I0129 00:40:50.150663       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="7.605704ms" userAgent="kube-probe/1.26" audit-ID="016c7efb-d7d8-4805-912d-8077c51072ac" srcIP="<IP_ADDRESS>:42738" resp=200
2024-01-29T01:40:51.208519929+01:00 E0129 00:40:51.208122       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
E0129 00:40:51.208232       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
2024-01-29T01:40:51.209724911+01:00 I0129 00:40:51.209368       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="47.166775ms" userAgent="Go-http-client/2.0" audit-ID="09fe7fe7-1fcd-4f6a-b54c-400f23fa1fc3" srcIP="<IP_ADDRESS>:46152" resp=200
2024-01-29T01:40:51.209827706+01:00 E0129 00:40:51.209511       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="09fe7fe7-1fcd-4f6a-b54c-400f23fa1fc3"
2024-01-29T01:40:51.209999648+01:00 E0129 00:40:51.209824       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:40:51.210543       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
2024-01-29T01:40:51.211369924+01:00 I0129 00:40:51.210934       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="49.624443ms" userAgent="Go-http-client/2.0" audit-ID="6d20c46c-0044-412f-b770-670c36988b24" srcIP="<IP_ADDRESS>:46152" resp=200
E0129 00:40:51.211076       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="6d20c46c-0044-412f-b770-670c36988b24"
E0129 00:40:51.211832       1 timeout.go:141] post-timeout activity - time-elapsed: 1.008782ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:51.212732738+01:00 I0129 00:40:51.212415       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="50.682076ms" userAgent="Go-http-client/2.0" audit-ID="a37f8fb3-eda5-41ea-935e-1a1afba2b570" srcIP="<IP_ADDRESS>:46152" resp=200
2024-01-29T01:40:51.212813218+01:00 E0129 00:40:51.212499       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
E0129 00:40:51.212520       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="a37f8fb3-eda5-41ea-935e-1a1afba2b570"
2024-01-29T01:40:51.214660987+01:00 E0129 00:40:51.214285       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0129 00:40:51.217872       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
I0129 00:40:51.219919       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="58.006266ms" userAgent="Go-http-client/2.0" audit-ID="e8ba5867-14d0-46fd-9fa9-2906543626a9" srcIP="<IP_ADDRESS>:46152" resp=200
E0129 00:40:51.220015       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="e8ba5867-14d0-46fd-9fa9-2906543626a9"
2024-01-29T01:40:51.220432755+01:00 E0129 00:40:51.220129       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:40:51.220609       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
2024-01-29T01:40:51.221692533+01:00 I0129 00:40:51.221406       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="40.523317ms" userAgent="Go-http-client/2.0" audit-ID="c5a1f8f0-ac25-475b-8d94-2e1eaeb2bd2a" srcIP="<IP_ADDRESS>:46152" resp=200
E0129 00:40:51.221505       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="c5a1f8f0-ac25-475b-8d94-2e1eaeb2bd2a"
E0129 00:40:51.221985       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
2024-01-29T01:40:51.224909708+01:00 E0129 00:40:51.224696       1 timeout.go:141] post-timeout activity - time-elapsed: 13.506933ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:51.225618680+01:00 E0129 00:40:51.225439       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:40:51.227997       1 timeout.go:141] post-timeout activity - time-elapsed: 15.401683ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:40:51.229043       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0129 00:40:51.229878       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:40:51.232022       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
2024-01-29T01:40:51.233526750+01:00 E0129 00:40:51.233408       1 timeout.go:141] post-timeout activity - time-elapsed: 13.329028ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:51.234424125+01:00 E0129 00:40:51.234314       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:40:51.235706       1 timeout.go:141] post-timeout activity - time-elapsed: 14.138405ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
I0129 00:40:52.820298       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="49.635276ms" userAgent="Go-http-client/2.0" audit-ID="aba6f90c-7801-499d-ad67-22004b1c1b48" srcIP="<IP_ADDRESS>:52136" resp=200
E0129 00:40:52.820604       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="aba6f90c-7801-499d-ad67-22004b1c1b48"
E0129 00:40:52.826529       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
E0129 00:40:52.829236       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
2024-01-29T01:40:52.830534835+01:00 I0129 00:40:52.830247       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="60.134976ms" userAgent="Go-http-client/2.0" audit-ID="d4b8dc48-08a7-46fc-8021-1399e01b9474" srcIP="<IP_ADDRESS>:52136" resp=200
2024-01-29T01:40:52.831279324+01:00 E0129 00:40:52.831066       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="d4b8dc48-08a7-46fc-8021-1399e01b9474"
E0129 00:40:52.831669       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
2024-01-29T01:40:52.833659030+01:00 E0129 00:40:52.832803       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
2024-01-29T01:40:52.833734270+01:00 E0129 00:40:52.832803       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
2024-01-29T01:40:52.833755529+01:00 I0129 00:40:52.833242       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="47.175923ms" userAgent="Go-http-client/2.0" audit-ID="b8f45365-b2cf-471b-970a-be716f3c6b1d" srcIP="<IP_ADDRESS>:52136" resp=200
2024-01-29T01:40:52.833773881+01:00 E0129 00:40:52.833451       1 timeout.go:141] post-timeout activity - time-elapsed: 1.504997ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:52.833817362+01:00 E0129 00:40:52.833438       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="b8f45365-b2cf-471b-970a-be716f3c6b1d"
2024-01-29T01:40:52.833842213+01:00 E0129 00:40:52.833556       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
I0129 00:40:52.833195       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="62.891788ms" userAgent="Go-http-client/2.0" audit-ID="00e9c757-5260-45d9-aab2-f1d085c0a345" srcIP="<IP_ADDRESS>:52136" resp=200
E0129 00:40:52.834152       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="00e9c757-5260-45d9-aab2-f1d085c0a345"
2024-01-29T01:40:52.834725774+01:00 E0129 00:40:52.834559       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0129 00:40:52.835994       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
I0129 00:40:52.838678       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="60.215048ms" userAgent="Go-http-client/2.0" audit-ID="8893de02-d005-4b80-b5d2-a1c9b043fc6f" srcIP="<IP_ADDRESS>:52136" resp=200
2024-01-29T01:40:52.839744479+01:00 E0129 00:40:52.839064       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
2024-01-29T01:40:52.839826811+01:00 E0129 00:40:52.839062       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="8893de02-d005-4b80-b5d2-a1c9b043fc6f"
E0129 00:40:52.841594       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:40:52.844413       1 timeout.go:141] post-timeout activity - time-elapsed: 9.949799ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:40:52.845382878+01:00 E0129 00:40:52.845144       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:40:52.847275       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
2024-01-29T01:40:52.848280836+01:00 E0129 00:40:52.848070       1 timeout.go:141] post-timeout activity - time-elapsed: 14.695009ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:40:52.850131       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:40:52.851222       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:40:52.853805       1 timeout.go:141] post-timeout activity - time-elapsed: 33.043963ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:40:52.855281       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:40:52.857867       1 timeout.go:141] post-timeout activity - time-elapsed: 18.684357ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
I0129 00:40:54.839687       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="57.18263ms" userAgent="k3s/v1.26.10+k3s1 (linux/arm64) kubernetes/a6687b5/system:serviceaccount:kube-system:generic-garbage-collector" audit-ID="685dc715-b1f0-4b33-b6ed-83efab697497" srcIP="<IP_ADDRESS>:52170" resp=200
I0129 00:40:55.055321       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="1.944749ms" userAgent="kube-probe/1.26" audit-ID="50b5a49f-1253-47ed-bd7c-810855e7c666" srcIP="<IP_ADDRESS>:42744" resp=200
I0129 00:40:58.411195       1 httplog.go:131] "HTTP" verb="GET" URI="/openapi/v2" latency="10.630142ms" userAgent="" audit-ID="71e2d5d7-4939-4445-9b61-8a2d62ffbbf9" srcIP="<IP_ADDRESS>:52170" resp=304
2024-01-29T01:40:58.413604438+01:00 I0129 00:40:58.412603       1 httplog.go:131] "HTTP" verb="GET" URI="/openapi/v2" latency="6.762549ms" userAgent="" audit-ID="53023b71-222b-4d34-9659-ea15345ac6b9" srcIP="<IP_ADDRESS>:58630" resp=304
I0129 00:40:58.415594       1 httplog.go:131] "HTTP" verb="GET" URI="/openapi/v2" latency="7.660796ms" userAgent="" audit-ID="ea3017e1-1627-4f35-9db6-bf11aa45a281" srcIP="<IP_ADDRESS>:50432" resp=304
I0129 00:40:58.418384       1 httplog.go:131] "HTTP" verb="GET" URI="/openapi/v2" latency="1.400498ms" userAgent="" audit-ID="f3f998a0-6974-4da8-bdc1-aa778a5b6c49" srcIP="<IP_ADDRESS>:52170" resp=304
I0129 00:40:58.420791       1 httplog.go:131] "HTTP" verb="GET" URI="/openapi/v2" latency="1.361184ms" userAgent="" audit-ID="b07dbaff-f331-47b8-aaaa-7238f022c5f5" srcIP="<IP_ADDRESS>:58630" resp=304
I0129 00:40:58.467176       1 httplog.go:131] "HTTP" verb="GET" URI="/openapi/v2" latency="1.409257ms" userAgent="" audit-ID="c5b5b5cd-19f9-4fe1-a780-b7a81341aa92" srcIP="<IP_ADDRESS>:50432" resp=304
I0129 00:41:00.140320       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="3.656817ms" userAgent="kube-probe/1.26" audit-ID="a22a8718-5e45-43a6-a59c-e01ff3303c47" srcIP="<IP_ADDRESS>:38916" resp=200
I0129 00:41:03.220173       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="51.806745ms" userAgent="k3s/v1.26.10+k3s1 (linux/arm64) kubernetes/a6687b5/controller-discovery" audit-ID="9ff8f08e-6a54-493c-b528-12e19184aabc" srcIP="<IP_ADDRESS>:52170" resp=200
E0129 00:41:03.532926       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
2024-01-29T01:41:03.533749797+01:00 E0129 00:41:03.533065       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
2024-01-29T01:41:03.534148791+01:00 I0129 00:41:03.533682       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="50.624873ms" userAgent="Go-http-client/2.0" audit-ID="dddedc55-0f38-410d-924b-79701913163c" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:41:03.534222087+01:00 E0129 00:41:03.533973       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="dddedc55-0f38-410d-924b-79701913163c"
2024-01-29T01:41:03.534428769+01:00 E0129 00:41:03.534163       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
2024-01-29T01:41:03.534757375+01:00 E0129 00:41:03.534571       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
I0129 00:41:03.534936       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="51.933836ms" userAgent="Go-http-client/2.0" audit-ID="89391d06-244f-46a1-a38d-4aa95076ab6d" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:41:03.535284016+01:00 E0129 00:41:03.535046       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="89391d06-244f-46a1-a38d-4aa95076ab6d"
2024-01-29T01:41:03.535740528+01:00 E0129 00:41:03.535505       1 timeout.go:141] post-timeout activity - time-elapsed: 1.434738ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:41:03.538646       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
I0129 00:41:03.547456       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="53.179485ms" userAgent="Go-http-client/2.0" audit-ID="c59307c7-2a9d-45af-b180-1c6f52b73a06" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:41:03.550178632+01:00 E0129 00:41:03.547962       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="c59307c7-2a9d-45af-b180-1c6f52b73a06"
2024-01-29T01:41:03.550228576+01:00 E0129 00:41:03.549539       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0129 00:41:03.551282       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:41:03.553204       1 timeout.go:141] post-timeout activity - time-elapsed: 17.033604ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:41:03.555984529+01:00 E0129 00:41:03.555796       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:41:03.558303       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:41:03.560119       1 timeout.go:141] post-timeout activity - time-elapsed: 11.857216ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
I0129 00:41:03.570826       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="88.962926ms" userAgent="Go-http-client/2.0" audit-ID="7cfa9e60-716c-44c0-919e-8bb3406802ab" srcIP="<IP_ADDRESS>:36818" resp=200
E0129 00:41:03.570935       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="7cfa9e60-716c-44c0-919e-8bb3406802ab"
I0129 00:41:03.578924       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="96.897792ms" userAgent="Go-http-client/2.0" audit-ID="6dd91c9a-d9bf-448c-a493-52251b15df67" srcIP="<IP_ADDRESS>:36818" resp=200
2024-01-29T01:41:03.580183806+01:00 E0129 00:41:03.579055       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="6dd91c9a-d9bf-448c-a493-52251b15df67"
E0129 00:41:03.580937       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0129 00:41:03.581140       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:41:03.582776       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:41:03.584526       1 timeout.go:141] post-timeout activity - time-elapsed: 13.411286ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:41:03.589700       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
2024-01-29T01:41:03.590029033+01:00 E0129 00:41:03.589767       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 00:41:03.591067       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:41:03.593982       1 timeout.go:141] post-timeout activity - time-elapsed: 14.780006ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
I0129 00:41:05.052404       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="2.977605ms" userAgent="kube-probe/1.26" audit-ID="7cc1ba8e-69c4-41ab-98ce-1a873eab17ed" srcIP="<IP_ADDRESS>:38924" resp=200
I0129 00:41:08.577825       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="50.294415ms" userAgent="k3s/v1.26.10+k3s1 (linux/arm64) kubernetes/a6687b5/system:serviceaccount:kube-system:resourcequota-controller" audit-ID="3f1ef2ad-6929-4012-b04b-50eb956bf92b" srcIP="<IP_ADDRESS>:52170" resp=200
I0129 00:41:10.139281       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="3.125362ms" userAgent="kube-probe/1.26" audit-ID="4d9adf99-c188-4955-91ac-60401a510a58" srcIP="<IP_ADDRESS>:39308" resp=200
I0129 00:41:15.058830       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="9.128904ms" userAgent="kube-probe/1.26" audit-ID="b1ae8689-4187-41a2-8f71-f2256280142d" srcIP="<IP_ADDRESS>:39318" resp=200
I0129 00:41:20.141532       1 httplog.go:131] "HTTP" verb="GET" URI="/healthz" latency="6.289352ms" userAgent="kube-probe/1.26" audit-ID="820dccc3-603c-4e8f-a367-db6fb1f9d3ad" srcIP="<IP_ADDRESS>:34340" resp=200
E0129 00:41:21.229184       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
2024-01-29T01:41:21.230241090+01:00 E0129 00:41:21.230020       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0129 00:41:21.230656       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
I0129 00:41:21.230907       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="69.848056ms" userAgent="Go-http-client/2.0" audit-ID="bd524e7e-fe2a-4f5b-973a-6d06fa093e84" srcIP="<IP_ADDRESS>:46152" resp=200
E0129 00:41:21.230990       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="bd524e7e-fe2a-4f5b-973a-6d06fa093e84"
2024-01-29T01:41:21.232883811+01:00 E0129 00:41:21.231196       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
E0129 00:41:21.232049       1 timeout.go:141] post-timeout activity - time-elapsed: 963.56µs, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
2024-01-29T01:41:21.232945421+01:00 I0129 00:41:21.232224       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="67.204095ms" userAgent="Go-http-client/2.0" audit-ID="6829aeb9-b0eb-4bf3-b8de-5a0318a3464a" srcIP="<IP_ADDRESS>:46152" resp=200
E0129 00:41:21.232289       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="6829aeb9-b0eb-4bf3-b8de-5a0318a3464a"
2024-01-29T01:41:21.232980143+01:00 I0129 00:41:21.232535       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="67.432369ms" userAgent="Go-http-client/2.0" audit-ID="4f64a548-3e46-4d9c-8383-28310af6145f" srcIP="<IP_ADDRESS>:46152" resp=200
E0129 00:41:21.232590       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="4f64a548-3e46-4d9c-8383-28310af6145f"
E0129 00:41:21.242175       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0129 00:41:21.248786       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0129 00:41:21.256005       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:41:21.259546       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
I0129 00:41:21.259828       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="50.465098ms" userAgent="Go-http-client/2.0" audit-ID="f0c5f977-6010-47b5-b8cf-657dfd6fad38" srcIP="<IP_ADDRESS>:46152" resp=200
E0129 00:41:21.262669       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="f0c5f977-6010-47b5-b8cf-657dfd6fad38"
E0129 00:41:21.259848       1 writers.go:118] apiserver was unable to write a JSON response: http2: stream closed
2024-01-29T01:41:21.263070797+01:00 E0129 00:41:21.260732       1 timeout.go:141] post-timeout activity - time-elapsed: 28.37655ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
I0129 00:41:21.262104       1 panic.go:838] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="72.474537ms" userAgent="Go-http-client/2.0" audit-ID="05cc1314-b02e-4578-a0bc-e52a178c0d0e" srcIP="<IP_ADDRESS>:46152" resp=200
2024-01-29T01:41:21.263436458+01:00 E0129 00:41:21.263304       1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="05cc1314-b02e-4578-a0bc-e52a178c0d0e"
E0129 00:41:21.262529       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0129 00:41:21.263640       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0129 00:41:21.270278       1 writers.go:131] apiserver was unable to write a fallback JSON response: http2: stream closed
E0129 00:41:21.273549       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0129 00:41:21.276973       1 timeout.go:141] post-timeout activity - time-elapsed: 44.166097ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0129 00:41:21.279643       1 timeout.go:141] post-timeout activity - time-elapsed: 19.741249ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>

@CC007 I am facing the same error, were you able to solve it?


Items contain duplicates if downloading with pool and filters

The current logic of dividing on batches by start_index and count doesn't account for filter.
When using filter, returned items _key don't correspond with actual index so the data repeats.

The solution is to use nodata https://github.com/scrapinghub/python-scrapinghub/pull/129