Reschedule jaeger-cassandra-schema-job upon failure

I'm using the production YAML file to deploy Jager :
root@kmaster01:~# kubectl  --namespace=opentracing get pod
NAME                                READY     STATUS             RESTARTS   AGE
cassandra-0                         1/1       Running            0          5m
cassandra-1                         1/1       Running            1          5m
cassandra-2                         1/1       Running            3          5m
jaeger-collector-1578691591-gm9dn   0/1       CrashLoopBackOff   3          6m
jaeger-query-1666845303-113hb       0/1       CrashLoopBackOff   2          6m

root@kmaster01:~# kubectl  --namespace=opentracing logs jaeger-collector-1578691591-gm9dn
2017/09/06 14:42:56 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:56 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:42:57 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1504708977.0914874,"caller":"collector/main.go:79","msg":"Unable to set up builder","error":"no connections were made when creating the session","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:208\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:79\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:122"}

root@kmaster01:~# kubectl  --namespace=opentracing logs jaeger-query-1666845303-113hb
2017/09/06 14:43:12 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/09/06 14:43:12 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1504708993.0407972,"caller":"query/main.go:71","msg":"Failed to init storage builder","error":"no connections were made when creating the session","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:208\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:71\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:100"}

Any idea why i've got these errors ? Thanks.

TL;DR: For the first couple of minutes or so, it should be fine to have those errors. Kubernetes will eventually spin up new instances of the pods and it will eventually work.
Long version: When deploying everything, we can't[1] have an orderly deployment order. Basically, we need to deploy Cassandra, create the schema required by Jaeger, deploy the Collector and the Query services. What you are seeing is the result of the Collector/Query starting before the schema was created, and after Cassandra has been initialized. The "create schema" is a Kubernetes job (kubectl logs job/jaeger-cassandra-schema-job) which waits up until 1 minute for Cassandra to be ready. If it times out, Kubernetes will schedule a new run (IIRC, not 100% sure on that now).
1 - I think there's a new feature on newer versions of Kubernetes where we could add the create-schema job, but I need to check that

By the way: if those won't go away after a few minutes, please do post the logs for the create-schema job:
$ kubectl logs job/jaeger-cassandra-schema-job

By the way: if those won't go away after a few minutes, please do post the logs for the create-schema job:
$ kubectl logs job/jaeger-cassandra-schema-job

@jpkrohling ok thanks
root@kmaster01:~# kubectl --namespace=opentracing get job
NAME                          DESIRED   SUCCESSFUL   AGE
jaeger-cassandra-schema-job   1         0            26m
root@kmaster01:~# kubectl --namespace=opentracing logs job/jaeger-cassandra-schema-job
Found 10 pods, using pod/jaeger-cassandra-schema-job-ggw25
Error from server (BadRequest): container "jaeger-cassandra-schema" in pod "jaeger-cassandra-schema-job-ggw25" is not available

I will wait a few minutes.

26 minutes is way too long already :) It looks like I'm wrong and Kubernetes won't reschedule a new run for the failed create-schema job. Could you try to remove it and deploy the services again? Something like:
kubectl delete job/jaeger-cassandra-schema-job
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production/jaeger-production-template.yml

(the last command should fail on all services except the create-schema job)

Output :
job "jaeger-cassandra-schema-job" created
Error from server (AlreadyExists): services "cassandra" already exists
Error from server (AlreadyExists): statefulsets.apps "cassandra" already exists
Error from server (AlreadyExists): deployments.extensions "jaeger-collector" already exists
Error from server (AlreadyExists): services "jaeger-collector" already exists
Error from server (AlreadyExists): services "zipkin" already exists
Error from server (AlreadyExists): deployments.extensions "jaeger-query" already exists
Error from server (AlreadyExists): services "jaeger-query" already exists
root@kmaster01:~# kubectl --namespace=opentracing logs job/jaeger-cassandra-schema-job

system_traces  system_schema  system_auth  system  system_distributed

Generating the schema for the keyspace jaeger_v1_dc1 and datacenter dc1
Using template file /cassandra-schema/v001.cql.tmpl with parameters:
    mode = prod
    datacenter = dc1
    keyspace = jaeger_v1_dc1
    replication = {'class': 'NetworkTopologyStrategy', 'dc1': '2' }
    trace_ttl = 172800
    dependencies_ttl = 0

Looks like your collector and query are now working.
The issue is then to make sure the create-schema job can be rescheduled upon failure.

I had the same problem but deleting and recreating the job fixed it.

 Reschedule jaeger-cassandra-schema-job upon failure

@heppu , @nlamirault , would you be able to give the fix a try and confirm it works?

I have met the same problem. The version of jaeger I use is below:
root@ubuntu-k8s-3:~# docker images|grep jaeger
jaegertracing/jaeger-query                                 0.9                 2e85f0280681        11 days ago         41.9 MB
jaegertracing/jaeger-collector                             0.9                 ab916122f094        11 days ago         20 MB
jaegertracing/jaeger-cassandra-schema                      0.6                 27f8532f0ab3        2 months ago        386 MB
root@ubuntu-k8s-3:~# docker images|grep cassandra
cassandra                                                  3.11                b2a0d42e1e97        2 days ago          385 MB
jaegertracing/jaeger-cassandra-schema                      0.6                 27f8532f0ab3        2 months ago        386 MB

``

First I deploy cassandra pod with production/cassandra.yml. 10 min later  I use jaeger-production-template.yml to deploy jaeger backend.

Then I found jaeger backends pods are all not working:

default                    jaeger-collector-216026522-v49hz       0/1       CrashLoopBackOff   4          2m        <IP_ADDRESS>      ubuntu-k8s-3
default                    jaeger-query-2386461091-wsh3g          0/1       CrashLoopBackOff   4          2m        <IP_ADDRESS>      ubuntu-k8s-3

The log of collector and query are below:

root@ubuntu-k8s-3:~# docker logs 1554b653aa0f
{"level":"info","ts":1510020066.1924856,"caller":"healthcheck/handler.go:46","msg":"Health Check server started","http-port":14269}
2017/11/07 02:01:06 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/11/07 02:01:06 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1510020066.3941922,"caller":"collector/main.go:88","msg":"Unable to set up builder","error":"no connections were made when creating the session","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:88\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:142\nruntime.main\n\t/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/proc.go:185"}
root@ubuntu-k8s-3:~# docker logs 1e791f7ac916
{"level":"info","ts":1510020071.1615207,"caller":"healthcheck/handler.go:46","msg":"Health Check server started","http-port":16687}
2017/11/07 02:01:11 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
2017/11/07 02:01:11 error: failed to connect to <IP_ADDRESS>:9042 due to error: Keyspace 'jaeger_v1_dc1' does not exist
{"level":"fatal","ts":1510020071.37031,"caller":"query/main.go:92","msg":"Failed to init storage builder","error":"no connections were made when creating the session","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:92\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/main.go:135\nruntime.main\n\t/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/proc.go:185"}

According the advice of @jpkrohling,  I got the log of jaeger-cassandra-schema-job and it seems the job work unexpectedly:

kubectl logs job/jaeger-cassandra-schema-job
error: timed out waiting for the condition

I delete the job and recreate it. The restarting log is below:

kubectl logs  job/jaeger-cassandra-schema-job
system_traces  system_schema  system_auth  system  system_distributed
Generating the schema for the keyspace jaeger_v1_dc1 and datacenter dc1
Using template file /cassandra-schema/v001.cql.tmpl with parameters:
mode = prod
datacenter = dc1
keyspace = jaeger_v1_dc1
replication = {'class': 'NetworkTopologyStrategy', 'dc1': '2' }
trace_ttl = 172800
dependencies_ttl = 0
:3:OperationTimedOut: errors={'<IP_ADDRESS>': 'Client request timeout. See Session.execute_async'}, last_host=<IP_ADDRESS>
:13:InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot add type in unknown keyspace jaeger_v1_dc1"
:18:InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot add type in unknown keyspace jaeger_v1_dc1"
:24:InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot add type in unknown keyspace jaeger_v1_dc1"
:29:InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot add type in unknown keyspace jaeger_v1_dc1"
:54:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:68:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:83:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:100:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:117:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:136:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:156:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:162:InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot add type in unknown keyspace jaeger_v1_dc1"
:175:ConfigurationException: Keyspace jaeger_v1_dc1 doesn't exist
:179:InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace jaeger_v1_dc1 does not exist"
It seems that it still do not work normal.

I try to recreate jaeger backend. After that, the result give me a surprise that the jaeger backend runs ok:

default                    jaeger-collector-216026522-m0tpr       1/1       Running   0          9m        <IP_ADDRESS>      ubuntu-k8s-3
default                    jaeger-query-2386461091-lbgr0          1/1       Running   0          9m        <IP_ADDRESS>      ubuntu-k8s-3

The latest template should deploy a create-schema job that will reschedule itself upon failure. Could you please confirm you are using a version with this fix? You can get the job's JSON/YAML representation by running something like this:
$ kubectl get job jaeger-cassandra-schema-job -o yaml

Make sure the restartPolicy is set to OnFailure. If this is already the case, then I would need more information about your environment, to determine why it hasn't been restarted.
Also, looks like the create-schema job is still using the 0.6 image. I think it should still work, but I'm opening a PR to get this fixed.
About Collector/Query being in the running state even after the failures: I believe this is expected, but if there's something missing (a table, index, ...), it might fail at runtime.
What is really strange is that the schema job seemed to work first, and then fail with a Client request timeout. This seems to indicate that there was a network failure. Given that the Collector/Query are working, it means that the keyspace jaeger_v1_dc1 was created.

@jpkrohling
The jaeger-cassandra-schema-job info is below:
# kubectl get job jaeger-cassandra-schema-job -o yaml
apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: 2017-11-07T02:39:39Z
  labels:
    app: jaeger
    jaeger-infra: cassandra-schema-job
  name: jaeger-cassandra-schema-job
  namespace: default
  resourceVersion: "20137088"
  selfLink: /apis/batch/v1/namespaces/default/jobs/jaeger-cassandra-schema-job
  uid: e70e7ef7-c364-11e7-9343-0650cc001a5b
spec:
  activeDeadlineSeconds: 120
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: e70e7ef7-c364-11e7-9343-0650cc001a5b
  template:
    metadata:
      creationTimestamp: null
      labels:
        controller-uid: e70e7ef7-c364-11e7-9343-0650cc001a5b
        job-name: jaeger-cassandra-schema-job
      name: cassandra-schema
    spec:
      containers:
      - env:
        - name: MODE
          value: prod
        - name: DATACENTER
          value: dc1
        image: jaegertracing/jaeger-cassandra-schema:0.6
        imagePullPolicy: IfNotPresent
        name: jaeger-cassandra-schema
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: OnFailure
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  completionTime: 2017-11-07T02:39:51Z
  conditions:
  - lastProbeTime: 2017-11-07T02:39:51Z
    lastTransitionTime: 2017-11-07T02:39:51Z
    status: "True"
    type: Complete
  startTime: 2017-11-07T02:39:39Z
  succeeded: 1

@jpkrohling We just had a case where the said job failed and was not rescheduled.
Events:
  Type     Reason            Age   From            Message
  ----     ------            ----  ----            -------
  Normal   SuccessfulCreate  56m   job-controller  Created pod: jaeger-cassandra-schema-job-9qt9c
  Normal   SuccessfulDelete  54m   job-controller  Deleted pod: jaeger-cassandra-schema-job-9qt9c
  Warning  DeadlineExceeded  54m   job-controller  Job was active longer than specified deadline

$ kubectl --namespace jaeger get job jaeger-cassandra-schema-job -o yaml
apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: "2020-01-09T14:31:39Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: cronjob-cassandra-schema
    app.kubernetes.io/instance: jaeger
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jaeger-cassandra-schema-job
    app.kubernetes.io/part-of: jaeger
  name: jaeger-cassandra-schema-job
  namespace: jaeger
  ownerReferences:
  - apiVersion: jaegertracing.io/v1
    controller: true
    kind: Jaeger
    name: jaeger
    uid: a32328e8-32ec-11ea-988f-42010aa8011c
  resourceVersion: "3943"
  selfLink: /apis/batch/v1/namespaces/jaeger/jobs/jaeger-cassandra-schema-job
  uid: bf85d3d6-32ec-11ea-988f-42010aa8011c
spec:
  activeDeadlineSeconds: 120
  backoffLimit: 6
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: bf85d3d6-32ec-11ea-988f-42010aa8011c
  template:
    metadata:
      annotations:
        linkerd.io/inject: disabled
        prometheus.io/scrape: "false"
        sidecar.istio.io/inject: "false"
      creationTimestamp: null
      labels:
        controller-uid: bf85d3d6-32ec-11ea-988f-42010aa8011c
        job-name: jaeger-cassandra-schema-job
    spec:
      containers:
      - env:
        - name: CQLSH_HOST
          value: cassandra-cassandra-nodes.cassandra.svc.cluster.local
        - name: MODE
          value: prod
        - name: DATACENTER
          value: cassandra
        - name: CASSANDRA_USERNAME
        - name: CASSANDRA_PASSWORD
        image: jaegertracing/jaeger-cassandra-schema:1.16.0
        imagePullPolicy: IfNotPresent
        name: jaeger-cassandra-schema
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: OnFailure
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  conditions:
  - lastProbeTime: "2020-01-09T14:33:39Z"
    lastTransitionTime: "2020-01-09T14:33:39Z"
    message: Job was active longer than specified deadline
    reason: DeadlineExceeded
    status: "True"
    type: Failed
  failed: 1
  startTime: "2020-01-09T14:31:39Z"

No logs to share yet from the failed job, sorry, they might be gone.
But in that case we certainly want to reschedule the job, too, right?

The schema job is idempotent, so, it can be safely rescheduled.

The schema job is idempotent, so, it can be safely rescheduled.

Thanks for the quick reply @jpkrohling -- yes, we did that manually. But is there something we can do in this project here to make this happen automatically? Why did was restartPolicy: OnFailure not sufficient in this case? :-).

Why was restartPolicy: OnFailure not sufficient in this case? :-).

I guess that's indeed the key question. I would guess that OnFailure means when the container exits with a non-zero return code, which isn't the case here. A quick look at the docs yields no confirmation on that, though. If you'd be willing to help, could you try to confirm this theory?

Still rather new to Kubernetes, but I was reading up on the topic.
The following resources show that the topic of job termination underwent quite some evolution: https://github.com/kubernetes/kubernetes/issues/39578
https://github.com/kubernetes/kubernetes/issues/30243
https://github.com/kubernetes/enhancements/issues/298
https://github.com/kubernetes/community/pull/583/files
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
I now understood a few key things I believe:

A job does not have a RestartPolicy.
The RestartPolicy that we talk about in the context of a job actually is the corresponding pod's RestartPolicy.
Hence, a failed job is never restarted magically by some RestartPolicy!

That explains the observations above where the system got into a permanent error state that required human intervention.
But... how can a job permanently fail? A job is meant to drive something to completion, no matter what! Usually.
The "no matter what" can be limited via mainly two mechanisms: a maximum pod restart/retry count (.spec.backoffLimit) or -- interestingly -- via a global deadline which applies to the entire duration of the job, spanning potentially many pod restarts.
Those two mechanisms I think should be applied rather carefully because they contradict the common reason for why one chooses to use a job in the first place: a job is for "driving something to completion, no matter what" and now we have two brutal methods to abort this process, w/o an automatic strategy handling the 'failed' case.
I think that the main reason to apply either job abortion mechanism is to handle a critical problem that cannot self-heal over time, i.e. a problem that requires human intervention, such as component misconfiguration.
In our case here I think we just want to keep trying.
In conclusion I believe that job/jaeger-cassandra-schema-job should not have a deadline applied. It should simply re-schedule the corresponding pod until that succeeds, which is the reason for why we have chosen to use a job for that in the first place, right? :-)
That is, I think we want to remove spec.activeDeadlineSeconds: https://github.com/jaegertracing/jaeger-kubernetes/blob/39dc1369228952164bcdde2350ebc91cf00e36b0/production/cassandra.yml#L125

Worth considering:

With job .spec.template.spec.restartPolicy = "OnFailure" the Pod stays on the node, but the container is re-run if schema creation fails.
With job .spec.template.spec.restartPolicy = "Never" if schema creation fails then the container crashes the entire pod and then the Job controller starts a new Pod.

To me it seems the latter might even be more robust in this case here (is more likely to heal various kinds of transient problems).

Thanks for your detective work, @jgehrcke!

I think that the main reason to apply either job abortion mechanism is to handle a critical problem that cannot self-heal over time, i.e. a problem that requires human intervention, such as component misconfiguration.

Agree, but wouldn't you say, though, that a job that has been running for 24h is in a critical state and needs to be killed? It should certainly be retried, but how do we give up on a specific execution and start a new one? If we just remove activeDeadlineSeconds, wouldn't it just hang indefinitely in case it faces a deadlock?

:male_detective: :-)

It should certainly be retried, but how do we give up on a specific execution and start a new one? If we just remove activeDeadlineSeconds, wouldn't it just hang indefinitely in case it faces a deadlock?

Looks like my detective work pays off here, too.. I am really brand new to Kubernetes, but I have found this in the docs for jobs:

Note that both the Job spec and the Pod template spec within the Job have an activeDeadlineSeconds field. Ensure that you set this field at the proper level.

Sounds like we want to set a deadline on the pod itself (and not on the job) to make sure that things are actually retried (whereas the job itself manages the retries).
If you feel better doing so I can try adding this to the PR, or maybe we want to treat that separately?

Sounds like we want to set a deadline on the pod itself (and not on the job) to make sure that the individual attempt does not take longer than T (even if bad code runs into a deadlock), enabling the job to actually retry at a minimum frequency 1/T.

Just to confirm my understanding then: are you saying that if activeDeadlineSeconds is set at the PodSpec level, the pod will get killed and a new pod will be created, without failing the whole job? That sounds exactly what we need, then.

are you saying that if activeDeadlineSeconds is set at the PodSpec level, the pod will get killed and a new pod will be created, without failing the whole job? That sounds exactly what we need, then.

Yes.
Now it all makes a lot of sense. "Run to completion" seems to be the main purpose of jobs... "Run to Completion" is in the title of https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion
How is "Run to Completion" achieved? It's a little buried in the docs, but I think the key mechanism really is "When a Pod fails, then the Job controller starts a new Pod". Simple, right? :) This also justifies why a failed job itself is a bit of a delicate, weird state, contradicting the idea that a job cannot fail, but runs until it completes. Given that it makes perfect sense that there is no magic automatically handling a failed job. A failed job is an unexpected, critical error.

It seems appropriate then to have at both places: in the pod with a 2-minute deadline, and on the job with a really high value (1d?). We certainly don't want to keep retrying executions if we can't get a single clean completion in a whole day, right?

It seems appropriate then to have at both places: in the pod with a 2-minute deadline, and on the job with a really high value (1d?).

I know where you come from. But the exact motivation behind such a high timeout value is a little hard to put into words. And that is a sign for unnecessary complexity.

We certainly don't want to keep retrying executions if we can't get a single clean completion in a whole day, right?

I am not sure about that. Let's say after a day the job did not succeed. This could manifest in one of two ways:

the job is in failed state because of activeDeadlineSeconds set to 1 day
the job did simply not yet succeed

First of all, both cases require human intervention (we ended up in this state because there is no sane automatic treatment). Secondly, that intervention will happen regardless of whether it is (1) or (2).
If I was the person to handle this case I would prefer (2). Because all I need to do is look at the logs, find the cause, and maybe if it is a permanent dependency error I will fix that and don't have to touch the job at all (the next run would simply succeed). If it turns out that indeed the job is flawed I can decide to delete the job, and restart it after modification.

In other words: what's the advantage of stopping the retry loop?

I think one point of view is: it's nice to at some point see a clear, explicit failed somewhere -- so that its' easy to discover which part of the system does not behave as expected.
I choose to not care too much :) Happy to add a short activeDeadlineSeconds for the pod, and a long activeDeadlineSeconds for the job.

In other words: what's the advantage of stopping the retry loop?

Having an upper bound on what's acceptable helps people not familiar with Jaeger to determine whether something is "still working", or if it's broken. If a schema job can't finish after 1 day worth of retries, then something is definitely wrong.

Happy to add a short activeDeadlineSeconds for the pod, and a long activeDeadlineSeconds for the job.

I have updated #125 with that!


underline error when i call it twice

Hi @davamp !
Thank you for using BFPaperTabBarController!
Could you please provide more information and some code that I can look at and try to debug?
Thanks!


[ENH] Swap transformer used in SummaryClassifier

Uses the collection transformer summary transformer instead of the series transformer. This should be more efficient with time series collections.

need to fix get_test_params


Feature/cleanup

Issue
Closes #68

Looks like lib/charms/observability_libs/v1/kubernetes_service_patch.py is also unused.
Delete it too?


i extremly un sure about how to use this ,,

i mean ,, where should i put the psd file ? and where to put the psb file ? and when ? imean ,everything should be very clear atleast put a screenshoot ,or better video about how to do it ,please

You right click in the project window and create/custom 2d/psber. then you put that psber file in any folder or parent folder of your psd files. it will copy psd files to psb files.
[cid:d955e245-de97-45ac-a7db-c0b50a6bde9b]

From: muafakul<EMAIL_ADDRESS>Sent: Tuesday, August 11, 2020 8:48 AM
To: Steffenvy/PSBer<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: [Steffenvy/PSBer] i extremly un sure about how to use this ,, (#1)
i mean ,, where should i put the psd file ? and where to put the psb file ? and when ? imean ,everything should be very clear atleast put a screenshoot ,or better video about how to do it ,please
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/Steffenvy/PSBer/issues/1, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKY6T5JZNCBKWGFZ6T5SMTTSADSU3ANCNFSM4P2WO36A.


MHC improvements

Issue #, if available:
3979
Description of changes:

In cluster_manager.go, CreateWorkloadCluster() was waiting for all CP nodes to be ready.  Now it only waits for the CP endpoint to the ready, meaning just the first CP node is ready.
The CNI is installed as soon as the CP endpoint is ready.
The machine health checks are added right after the CNI is installed.

NOTE: These changes cause some of the unit tests to fail.  That needs to be fixed before merging.
ALSO NOTE: These changes do not address the failure of the first CP machine.  That needs to be fixed separately.
Testing (if applicable):
Manually tested by creating a cluster with three worker machines, one of which lacked a network connection.  The defective machine was detected by MHC and replaced by CAPI.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

No longer needed.


Problems when playing a SAN move with move number

When you call board.playMove("1. c4") the move is not played.
It should be possible to parse SAN moves that start with <spaces><numbers><spaces><dots><spaces>, or any combination of those elements with missing ones in order.

v5.3.8 fixes the issue.


Add cmark

Project URL
https://github.com/commonmark/cmark
Category
Libraries > C
Description
Add cmark, a C reference library to parse and render CommonMark.
Why it should be included to awesome-markdown (optional)
This library is actively maintained and seems to serve as a reference implementation of CommonMark parsing.
Checklist

[x] Only one project/change is in this pull request
[x] Addition in alphabetical order
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

          1 Warning

:warning:
Found 5 link issues, a project collaborator will take care of these, thanks :)

Link issues by awesome_bot

Line
Status
Link

169
522
https://dillinger.io/

173
Error
https://markable.in/  SSL_connect returned=1 errno=0 state=error: certificate verify failed (ok)

176
Error
https://www.tablesgenerator.com/markdown_tables  SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

179
Error
https://socrates.io/  SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)

200
502
https://gitprint.com/

  Generated by :no_entry_sign: Danger

Huge thanks @BubuAnabelas for the incredible fast merge! Pinging @jgm to let him know about this PR ! 😃

@kdeldycke thank you for the pull requests!


Always enable app engines in prod builds

Purpose

Summary of Changes

Side Effects

Feature Flags

QA Notes

Ticket

https://openscience.atlassian.net/browse/EMB-
Reviewer Checklist

[ ] meets requirements
[ ] easy to understand
[ ] DRY
[ ] testable and includes test(s)
[ ] changes described in CHANGELOG.md

Pull Request Test Coverage Report for Build 1585

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 66.416%

Totals

Change from base Build 1579:
0.0%

Covered Lines:
3179

Relevant Lines:
4423

💛  - Coveralls


improve 417: correct the response body for PUT submission

I have changed the sql so that it returns reviewState and updatedAt. Also removed the readable attribute from current field.
In lib/model/frames/submission.js file, I see several HACK/TODO tags around constructor of submission. @issa-tseng can you please share some details around that, what should be the ideal way to create submission object?

hmm, i’ve reread the file several times and did not find a TODO comment in it? am i looking at the wrong version?

On Apr 23, 2022, at 10:29, Sadiq Khoja @.***> wrote:

I have changed the sql so that it returns reviewState and updatedAt. Also removed the readable attribute from current field.
In lib/model/frames/submission.js file, I see several HACK/TODO tags around constructor of submission. @issa-tseng can you please share some details around that, what should be the ideal way to create submission object?
You can view, comment on, or merge this pull request online at:
https://github.com/getodk/central-backend/pull/475
Commit Summary
9a324a6 improve 417: correct the response body for PUT submission
File Changes  (2 files)
M lib/model/frames/submission.js (2)
M lib/model/query/submissions.js (11)
Patch Links:
https://github.com/getodk/central-backend/pull/475.patch
https://github.com/getodk/central-backend/pull/475.diff
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

@issa-tseng  my bad, it's lib/model/query/submissions.js

hey @sadiqkhoja the comment doesn't have to do with the constructor so much as we usually have the database return us its complete current data for the object at the end of the operation and return that as the result, but here we are instead copying in memory values that we believe the database has inserted.

Hi @sadiqkhoja, we haven't had a chance to review this PR yet, but we're hoping to take a look soon. Thanks for looking into this issue!

Hi @sadiqkhoja! I wanted to jump in and introduce myself -- I'm Kathleen, part of the core Central team! I wasn't able to look into this particular PR and issue and wrap my head around it until now, but here's what I'm thinking:
I've been looking at the API responses for submission versions (when the contents of a submission is updated and you get access to the previous versions). This PUT request is how a submission gets updated/edited, and the response actually matches what is returned by ../submissions/<instanceId>/versions and ../submissions/<instanceId>/versions/<instanceId>. It's not a full Submission, it's like a snapshot of a Submission, what internally in the code is called a SubmissionDef.
Anyway, I think (and have discussed with @matthew-white) that the original issue #417 is not the problem after all, and the real issue is that the documentation is wrong and describes a full Submission response in places where we don't actually return that.
I wrote this test to just print what the different requests look like. If you'll notice, the first response and the last match!
I have some specific comments on your code changes. I think your approach was absolutely in the right direction for what the issue said, and there is still a change you could make to improve the clarity of the code. I hope it is still a good learning experience for you -- it is for me!
it.only('should return PUT response in correct format', testService((service) =>
  service.login('alice', (asAlice) =>
    asAlice.post('/v1/projects/1/forms/simple/submissions')
      .send(testData.instances.simple.one)
      .set('Content-Type', 'application/xml')
      .expect(200)
      .then(() => service.login('bob', (asBob) =>
        asBob.put('/v1/projects/1/forms/simple/submissions/one')
          .set('Content-Type', 'text/xml')
          .send(withSimpleIds('one', 'two'))
          .expect(200)
          .then(({ body }) => {
            console.log("put response in question");
            console.log(body);
            //body.should.be.a.Submission(); // assertion fails
            body.should.be.a.SubmissionDef();
          })))
      .then(() => asAlice.get('/v1/projects/1/forms/simple/submissions/one')
        .expect(200)
        .then(({ body }) => {
          console.log("what it a submission looks like");
          console.log(body);
        }))
      .then(() => asAlice.get('/v1/projects/1/forms/simple/submissions/one/versions')
        .expect(200)
        .then(({ body }) => {
          console.log("what list of versions looks like");
          console.log(body);
        }))
      .then(() => asAlice.get('/v1/projects/1/forms/simple/submissions/one/versions/two')
        .expect(200)
        .then(({ body }) => {
          console.log("what it specific version of a submission looks like");
          console.log(body);
        })))));

  api: /forms/:id/submissions
    /:instanceId PUT
::ffff:<IP_ADDRESS> - - [06/May/2022:16:39:48 +0000] "POST /v1/sessions HTTP/1.1" 200 228
::ffff:<IP_ADDRESS> - - [06/May/2022:16:39:48 +0000] "POST /v1/projects/1/forms/simple/submissions HTTP/1.1" 200 127
::ffff:<IP_ADDRESS> - - [06/May/2022:16:39:48 +0000] "POST /v1/sessions HTTP/1.1" 200 228
::ffff:<IP_ADDRESS> - - [06/May/2022:16:39:48 +0000] "PUT /v1/projects/1/forms/simple/submissions/one HTTP/1.1" 200 162
put response in question
{
  instanceId: 'two',
  submitterId: 6,
  createdAt: '2022-05-06T16:39:48.511Z',
  instanceName: null,
  current: true,
  deviceId: null,
  userAgent: 'node-superagent/3.8.3'
}
::ffff:<IP_ADDRESS> - - [06/May/2022:16:39:48 +0000] "GET /v1/projects/1/forms/simple/submissions/one HTTP/1.1" 200 153
what it a submission looks like
{
  instanceId: 'one',
  submitterId: 5,
  deviceId: null,
  createdAt: '2022-05-06T16:39:48.489Z',
  updatedAt: '2022-05-06T16:39:48.511Z',
  reviewState: 'edited'
}
::ffff:<IP_ADDRESS> - - [06/May/2022:16:39:48 +0000] "GET /v1/projects/1/forms/simple/submissions/one/versions HTTP/1.1" 200 328
what list of versions looks like
[
  {
    submitterId: 6,
    createdAt: '2022-05-06T16:39:48.511Z',
    instanceName: null,
    instanceId: 'two',
    current: true,
    deviceId: null,
    userAgent: 'node-superagent/3.8.3'
  },
  {
    submitterId: 5,
    createdAt: '2022-05-06T16:39:48.489Z',
    instanceName: null,
    instanceId: 'one',
    current: false,
    deviceId: null,
    userAgent: 'node-superagent/3.8.3'
  }
]
::ffff:<IP_ADDRESS> - - [06/May/2022:16:39:48 +0000] "GET /v1/projects/1/forms/simple/submissions/one/versions/two HTTP/1.1" 200 162
what it specific version of a submission looks like
{
  submitterId: 6,
  createdAt: '2022-05-06T16:39:48.511Z',
  instanceName: null,
  instanceId: 'two',
  current: true,
  deviceId: null,
  userAgent: 'node-superagent/3.8.3'
}
      ✔ should return PUT response in correct format (342ms)

@sadiqkhoja and @ktuite, could you two take another look at this PR as part of v2023.1? Making small code changes and/or adding comments if that still seems useful, or otherwise closing out the PR.

@sadiqkhoja, now that you've made the changes in #783, do you think this PR can be closed? Is there anything else from this PR that we should capture?


Error when pushing artifacts: "BrokenPipeError: [Errno 32] Broken pipe"

In GitLab by @jjardon on Jul 25, 2017, 10:53
https://gitlab.com/baserock/definitions/-/jobs/24166576

In GitLab by @juergbi on Jul 26, 2017, 13:41
mentioned in commit 9b54d40ec8267fef8c3e1a75729c0bf6ba34579d

In GitLab by @juergbi on Jul 26, 2017, 13:41
mentioned in merge request !66

In GitLab by @tristanvb on Jul 26, 2017, 13:51
closed via merge request !66


java.net.ProtocolException: Control frames must be final

Hey there - I am building a sample app for MobileNativeFoundation/Store and dropbox/componentbox. I can connect to our Socket.IO server using Postman but am hitting connection errors on Android. Can you please take a look?
Problem

Cannot connect to Socket.IO server on Android

Related To

#21

Relevant Code

https://github.com/matt-ramotar/retriever-xplat/commit/744f5167d4f7a51861748096e9880dd08a361a2d

Reproduction

Clone https://github.com/matt-ramotar/retriever-xplat
Build and run
Log in with demo user
View logs

+1

@Alex009 news?

@Alex009 any update here?

maybe #31 that fixed in 0.6.0 will fix this too


support for IHostBuilder configuration missing.

Please correct me if I'm wrong. Seems that saturn is IWebHostBuilder centric. Since .NET 3.1 there is a generic host interface IHostBuilder to use instead. So there is no possibility to access the IHostBuilder interface in Saturn?
An example where it would be already useful is the Orleans 3.0 co-hosting in one process together with a web application. The UseOrleans(this IHostBuilder) extension method is defined for IHostBuilder rather than IWebHostBuilder where the most part of other web-related configuration is kept.
I thouthg first to come with a PR, adding additional host_config overload, but failed to make it compilable.
Or may be I miss an obvious way around saturn how this could be easily accomplished?

Implemented as part of #257


GitHub links on program pages not working as intended

The GitHub link on the program page is opening a new window with the current page's content instead of going to the program's github repo.

complete please review

All good


[BEAM-2570] Fix breakage after cloud-bigquery updated

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify.
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Fix BigqueryTornadoesIT failure when google-cloud-bigquery updated to 0.25.0. The new version changes query.fetch_data() return type.
Test is done by locally running BigqueryTornadoesIT against Dataflow service.

+R: @aaltay

retest this please

Coverage decreased (-0.01%) to 70.955% when pulling 822f4cd46758afa5dab5ee4e8dafdf42eee64dc8 on markflyhigh:update-cloud-bigquery-lib into 56e4251deeb080ceff331fc4adb5d68609b04c71 on apache:master.

LGTM. Will merge after tests pass.

Run Python PostCommit

Coverage decreased (-0.02%) to 70.941% when pulling 64e49c5a946324e8d5391211e1880663eb917b42 on markflyhigh:update-cloud-bigquery-lib into 56e4251deeb080ceff331fc4adb5d68609b04c71 on apache:master.


The Top Menu "File" shows options for "edit" & edit is gone

In the latest version, 2.2.2 released here the top menu shows only "file" but no "edit". When clicking on "file" I see all the typical options for edit.
Screenshot:

I have a feeling https://github.com/nylas-mail-lives/nylas-mail/releases/tag/2.2.2-4alpha resolves this issue

@austriker27 Did you try the above release? Did that work?


[LLVMGPU] Pad to intrinsic shape in LLVMGPUPadAndVectorDistribute pipeline

This patch makes LLVMGPUPromoteToFitMMA pass pad to a multiple of intrinsic shape, instead of padding to 1.
Fixes https://github.com/iree-org/iree/issues/18602

Can you add a test to https://github.com/iree-org/iree/blob/main/compiler/src/iree/compiler/Codegen/LLVMGPU/test/promote_matmul_to_fit_mma.mlir ?

I modified the first test to use lowering_config to decide tile sizes multiples.

cc @nirvedhmeshram


Correlating color to the warping direction

Red color corresponds to a positive warping in the 2D out-of-plane plot, and blue to a negative warping.

Excellent. Bug fixed.