pipeline's job with empty_files: true fails if input repo has a lot of files

What happened?:
job fails with failure: datum failed  when I try to use repo with > 1M files as input for pipeline.
Example of pipeline:
cat pull-data.pipeline
{
  "pipeline": {
    "name": "test-spreading-data"
  },
  "description": "This pipeline is going to",
  "transform": {
    "cmd": [
      "/bin/bash"
    ],
    "stdin": [
      "ls -laR /pfs/in/"
    ],
    "image": "ds-etl-image:some-tag",
    "debug": true
  },
  "input": {
    "pfs": {
      "name": "in",
      "repo": "add-some-mappings",
      "glob": "/",
      "empty_files": true
    }
  }
}

How to reproduce it (as minimally and precisely as possible)?:

create a repo with many small files. In my case I have > 1M files
create and run pipline like above

Anything else we need to know?:
 ➜  pachctl list job --no-pager
ID                               PIPELINE            STARTED       DURATION           RESTART PROGRESS  DL       UL       STATE
2460d4d9b7a54d1689b78268ad6de620 test-spreading-data 5 minutes ago 3 minutes          0       0 + 0 / 1 0B       0B       failure: datum failed

➜  pachctl logs --job 2460d4d9b7a54d1689b78268ad6de620 --worker
started datum task
transform worker datum task: job_id:"2460d4d9b7a54d1689b78268ad6de620" datums_object:"artifacts/job-2460d4d9b7a54d1689b78268ad6de620/chunk-datum-list-47aaef043278461892e3a54b5d42dcf1" output_commit:<repo:<name:"test-spreading-data" > id:"aad7bdb941b74f4b9fb6905a6b0b79a7" >
started processing datums
starting to download data
errored downloading data after 1m11.734806163s: open /pfs/.scratch/7309da433c8e4eef917036df2ba4d048/in/059152968-f6vMmLjNZM.json: no space left on device
failed processing datum: open /pfs/.scratch/7309da433c8e4eef917036df2ba4d048/in/059152968-f6vMmLjNZM.json: no space left on device, retrying in 0s
starting to download data
errored downloading data after 1m7.134327738s: open /pfs/.scratch/668e77b7bda94761b82d213f425776c5/in/059152967-f6vMmLjNZM.json: no space left on device
failed processing datum: open /pfs/.scratch/668e77b7bda94761b82d213f425776c5/in/059152967-f6vMmLjNZM.json: no space left on device, retrying in 0s
starting to download data
errored downloading data after 1m5.569215947s: open /pfs/.scratch/55b95bc296b04a74b4a7d0da3686db96/in/059152964-f6vMmLjNZM.json: no space left on device
failed to process datum with error: open /pfs/.scratch/55b95bc296b04a74b4a7d0da3686db96/in/059152964-f6vMmLjNZM.json: no space left on device
github.com/pachyderm/pachyderm/src/client/pkg/errors.EnsureStack
        /app/src/client/pkg/errors/errors.go:50
github.com/pachyderm/pachyderm/src/server/worker/driver.(*driver).downloadData
        /app/src/server/worker/driver/driver.go:631
github.com/pachyderm/pachyderm/src/server/worker/driver.(*driver).WithData
        /app/src/server/worker/driver/driver.go:469
github.com/pachyderm/pachyderm/src/server/worker/pipeline/transform.processDatum.func2
        /app/src/server/worker/pipeline/transform/worker.go:452
github.com/pachyderm/pachyderm/src/server/pkg/backoff.RetryUntilCancel
        /app/src/server/pkg/backoff/retry.go:113
github.com/pachyderm/pachyderm/src/server/worker/pipeline/transform.processDatum
        /app/src/server/worker/pipeline/transform/worker.go:448
github.com/pachyderm/pachyderm/src/server/worker/pipeline/transform.handleDatumTask.func<IP_ADDRESS>.1
        /app/src/server/worker/pipeline/transform/worker.go:311
golang.org/x/sync/errgroup.(*Group).Go.func1
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1374
finished processing datums
finished datum task

➜  kubectl logs pipeline-test-spreading-data-v1-qc4qb -c user
{"pipelineName":"test-spreading-data","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:17.539670781Z","message":"started transform spawner process"}
{"pipelineName":"test-spreading-data","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:17.543937385Z","message":"transform spawner started"}
{"pipelineName":"test-spreading-data","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:17.744860717Z","message":"created new job \"2460d4d9b7a54d1689b78268ad6de620\" for output commit \"aad7bdb941b74f4b9fb6905a6b0b79a7\""}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:17.756543882Z","message":"started waiting for job inputs"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:17.761892829Z","message":"finished waiting for job inputs"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:17.761947938Z","message":"started constructing datum iterator"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:24.710939497Z","message":"finished constructing datum iterator"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:24.711021841Z","message":"updating job info, state: JOB_RUNNING"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:24.726882622Z","message":"started processing job datums"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:24.726943349Z","message":"processJobRunning creating task channel"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:24.727008126Z","message":"started collecting datums for tasks"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:24.727008126Z","message":"started collecting datums for tasks"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:18:24.770762740Z","message":"finished collecting datums for tasks"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","ts":"2021-02-25T13:18:24.787865800Z","message":"started datum task"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","ts":"2021-02-25T13:18:24.788427934Z","message":"transform worker datum task: job_id:\"2460d4d9b7a54d1689b78268ad6de620\" datums_object:\"artifacts/job-2460d4d9b7a54d1689b78268ad6de620/chunk-datum-list-47aaef043278461892e3a54b5d42dcf1\" output_commit:\u003crepo:\u003cname:\"test-spreading-data\" \u003e id:\"aad7bdb941b74f4b9fb6905a6b0b79a7\" \u003e "}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","ts":"2021-02-25T13:18:24.788466121Z","message":"started processing datums"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:18:24.826251842Z","message":"starting to download data"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:19:36.561261391Z","message":"errored downloading data after 1m11.734806163s: open /pfs/.scratch/7309da433c8e4eef917036df2ba4d048/in/059152968-f6vMmLjNZM.json: no space left on device"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:19:36.561399673Z","message":"failed processing datum: open /pfs/.scratch/7309da433c8e4eef917036df2ba4d048/in/059152968-f6vMmLjNZM.json: no space left on device, retrying in 0s"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:19:36.561451803Z","message":"starting to download data"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:20:43.695872664Z","message":"errored downloading data after 1m7.134327738s: open /pfs/.scratch/668e77b7bda94761b82d213f425776c5/in/059152967-f6vMmLjNZM.json: no space left on device"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:20:43.695980739Z","message":"failed processing datum: open /pfs/.scratch/668e77b7bda94761b82d213f425776c5/in/059152967-f6vMmLjNZM.json: no space left on device, retrying in 0s"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:20:43.696053128Z","message":"starting to download data"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:21:49.265358674Z","message":"errored downloading data after 1m5.569215947s: open /pfs/.scratch/55b95bc296b04a74b4a7d0da3686db96/in/059152964-f6vMmLjNZM.json: no space left on device"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","datumId":"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108","data":[{"path":"/","hash":"2GSARa+4MLXCVSZSHwv8YH4eA/6z8+S4f0Eu953qrNo="}],"ts":"2021-02-25T13:21:49.265523808Z","message":"failed to process datum with error: open /pfs/.scratch/55b95bc296b04a74b4a7d0da3686db96/in/059152964-f6vMmLjNZM.json: no space left on device\ngithub.com/pachyderm/pachyderm/src/client/pkg/errors.EnsureStack\n\t/app/src/client/pkg/errors/errors.go:50\ngithub.com/pachyderm/pachyderm/src/server/worker/driver.(*driver).downloadData\n\t/app/src/server/worker/driver/driver.go:631\ngithub.com/pachyderm/pachyderm/src/server/worker/driver.(*driver).WithData\n\t/app/src/server/worker/driver/driver.go:469\ngithub.com/pachyderm/pachyderm/src/server/worker/pipeline/transform.processDatum.func2\n\t/app/src/server/worker/pipeline/transform/worker.go:452\ngithub.com/pachyderm/pachyderm/src/server/pkg/backoff.RetryUntilCancel\n\t/app/src/server/pkg/backoff/retry.go:113\ngithub.com/pachyderm/pachyderm/src/server/worker/pipeline/transform.processDatum\n\t/app/src/server/worker/pipeline/transform/worker.go:448\ngithub.com/pachyderm/pachyderm/src/server/worker/pipeline/transform.handleDatumTask.func<IP_ADDRESS>.1\n\t/app/src/server/worker/pipeline/transform/worker.go:311\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","ts":"2021-02-25T13:21:49.265604387Z","message":"finished processing datums"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","ts":"2021-02-25T13:21:49.265696091Z","message":"finished datum task"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.279843962Z","message":"updating job info, state: JOB_RUNNING"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.290567649Z","message":"finished running datum tasks"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.406042197Z","message":"processJobRunning updating task to merging, total stats: process_stats:\u003cdownload_time:\u003cseconds:65 nanos:569419431 \u003e process_time:\u003c\u003e upload_time:\u003c\u003e \u003e datums_failed:1 failed_datum_id:\"acbec1eb8f6c26f6edb5b2c890d2293559c8ee2f7900ce9fa5e5e902fe210108\" "}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.406118079Z","message":"updating job info, state: JOB_MERGING"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.416258642Z","message":"finished processing job datums"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.416327145Z","message":"started merging job hashtrees"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.416350325Z","message":"sending out 0 merge tasks"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.417910351Z","message":"merge results: [\u003cnil\u003e] trees (0 bytes), [\u003cnil\u003e] stats trees (0 bytes)"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.418000550Z","message":"failing job with reason: datum failed"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.498648201Z","message":"killing job with reason: output commit closed"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.498740925Z","message":"Cleaning job artifacts"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:21:49.504055998Z","message":"Cleaning job artifacts"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:22:41.440471036Z","message":"finished merging job hashtrees"}
{"pipelineName":"test-spreading-data","jobId":"2460d4d9b7a54d1689b78268ad6de620","workerId":"pipeline-test-spreading-data-v1-qc4qb","master":true,"ts":"2021-02-25T13:22:41.440572007Z","message":"master done running processJobs"}

Storage container errors:
2021-02-25T13:18:24Z ERROR pfs.BlockAPI.Obj.InspectTag {"duration":0.022992257,"error":"tagGetter: tag 5b46af407b7980697697bd205f7edd65dc4ed1a8e41fa3b8ff86634e525f41d0732b not found","request":{"name":"5b46af407b7980697697bd205f7edd65dc4ed1a8e41fa3b8ff86634e525f41d0732b"},"response":null,"stack":["github.com/pachyderm/pachyderm/src/server/pfs/server.(*objBlockAPIServer).tagGetter\n\t/app/src/server/pfs/server/obj_block_api_server.go:1233","github.com/golang/groupcache.GetterFunc.Get\n\t/go/pkg/mod/github.com/golang/groupcache@v0.0.0-20191027212112-611e8accdfc9/groupcache.go:54","github.com/golang/groupcache.(*Group).getLocally\n\t/go/pkg/mod/github.com/golang/groupcache@v0.0.0-20191027212112-611e8accdfc9/groupcache.go:296","github.com/golang/groupcache.(*Group).load.func1\n\t/go/pkg/mod/github.com/golang/groupcache@v0.0.0-20191027212112-611e8accdfc9/groupcache.go:279","github.com/golang/groupcache/singleflight.(*Group).Do\n\t/go/pkg/mod/github.com/golang/groupcache@v0.0.0-20191027212112-611e8accdfc9/singleflight/singleflight.go:56","github.com/golang/groupcache.(*Group).load\n\t/go/pkg/mod/github.com/golang/groupcache@v0.0.0-20191027212112-611e8accdfc9/groupcache.go:238","github.com/golang/groupcache.(*Group).Get\n\t/go/pkg/mod/github.com/golang/groupcache@v0.0.0-20191027212112-611e8accdfc9/groupcache.go:225","github.com/pachyderm/pachyderm/src/server/pfs/server.(*objBlockAPIServer).InspectTag\n\t/app/src/server/pfs/server/obj_block_api_server.go:859","github.com/pachyderm/pachyderm/src/client/pfs._ObjectAPI_InspectTag_Handler.func1\n\t/app/src/client/pfs/pfs.pb.go:8703","github.com/opentracing-contrib/go-grpc.OpenTracingServerInterceptor.func1\n\t/go/pkg/mod/github.com/opentracing-contrib/go-grpc@v0.0.0-20180928155321-4b5a12d3ff02/server.go:44","github.com/pachyderm/pachyderm/src/client/pfs._ObjectAPI_InspectTag_Handler\n\t/app/src/client/pfs/pfs.pb.go:8705","google.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024","google.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313","google.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722","runtime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"]} []

2021-02-25T13:21:49Z ERROR pfs.API.WalkFile {"duration":204.464132175,"error":"rpc error: code = Canceled desc = context canceled","request":{"file":{"commit":{"repo":{"name":"add-some-mappings"},"id":"1b7573cc3a554c52a18b943af1d67049"},"path":"/"}},"response":"response stream with 721120 objects","stack":null} []
2021-02-25T13:21:49Z INFO pfs.BlockAPI.Obj.PutObjDirect {"request":{"obj":"artifacts/job-2460d4d9b7a54d1689b78268ad6de620/hashtrees","value":"AAAAAAAAAAA="}} []
2021-02-25T13:21:49Z ERROR pfs.API.WalkFile {"duration":65.594260765,"error":"rpc error: code = Canceled desc = context canceled","request":{"file":{"commit":{"repo":{"name":"add-some-mappings"},"id":"1b7573cc3a554c52a18b943af1d67049"},"path":"/"}},"response":"response stream with 721120 objects","stack":null} []
2021-02-25T13:21:49Z ERROR pfs.API.WalkFile {"duration":132.728892555,"error":"rpc error: code = Canceled desc = context canceled","request":{"file":{"commit":{"repo":{"name":"add-some-mappings"},"id":"1b7573cc3a554c52a18b943af1d67049"},"path":"/"}},"response":"response stream with 721120 objects","stack":null} []

2021-02-25T13:21:49Z ERROR transaction.API.BatchTransaction {"duration":0.00139533,"error":"error running request 1 of 2: commit aad7bdb941b74f4b9fb6905a6b0b79a7 in repo test-spreading-data has already finished","request":{"requests":[{"finish_commit":{"commit":{"repo":{"name":"test-spreading-data"},"id":"aad7bdb941b74f4b9fb6905a6b0b79a7"},"empty":true}},{"update_job_state":{"job":{"id":"2460d4d9b7a54d1689b78268ad6de620"},"state":4,"reason":"output commit closed","data_failed":1,"data_total":1,"stats":{"download_time":{"seconds":65,"nanos":569419431},"process_time":{},"upload_time":{}}}}]},"response":null,"stack":["github.com/pachyderm/pachyderm/src/server/transaction/server.(*driver).runTransaction\n\t/app/src/server/transaction/server/driver.go:239","github.com/pachyderm/pachyderm/src/server/transaction/server.(*driver).batchTransaction.func1\n\t/app/src/server/transaction/server/driver.go:71","github.com/pachyderm/pachyderm/src/server/pkg/transactionenv.(*TransactionEnv).WithWriteContext.func1\n\t/app/src/server/pkg/transactionenv/env.go:394","github.com/pachyderm/pachyderm/src/server/pkg/collection.runSTM.func1\n\t/app/src/server/pkg/collection/transaction.go:109","runtime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"]} []
2021-02-25T13:21:49Z INFO pfs.API.FinishCommit {"request":{"commit":{"repo":{"name":"test-spreading-data"},"id":"aad7bdb941b74f4b9fb6905a6b0b79a7"},"empty":true}} []
2021-02-25T13:21:49Z ERROR pfs.API.FinishCommit {"duration":0.001280581,"error":"commit aad7bdb941b74f4b9fb6905a6b0b79a7 in repo test-spreading-data has already finished","request":{"commit":{"repo":{"name":"test-spreading-data"},"id":"aad7bdb941b74f4b9fb6905a6b0b79a7"},"empty":true},"response":null,"stack":null} []
2021-02-25T13:21:49Z INFO pps.API.UpdateJobState {"request":{"job":{"id":"2460d4d9b7a54d1689b78268ad6de620"},"state":4,"reason":"output commit closed","data_failed":1,"data_total":1,"stats":{"download_time":{"seconds":65,"nanos":569419431},"process_time":{},"upload_time":{}}}} []
2021-02-25T13:21:49Z ERROR pps.API.UpdateJobState {"duration":0.001713303,"error":"job 2460d4d9b7a54d1689b78268ad6de620 has already finished","request":{"job":{"id":"2460d4d9b7a54d1689b78268ad6de620"},"state":4,"reason":"output commit closed","data_failed":1,"data_total":1,"stats":{"download_time":{"seconds":65,"nanos":569419431},"process_time":{},"upload_time":{}}},"response":null,"stack":null} []

Environment?:

Kubernetes version (use kubectl version):

(.venv) ➜  tmp kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-dispatcher", GitCommit:"fd22db44e150011eccc8729db223945384460143", GitTreeState:"clean", BuildDate:"2020-07-24T07:27:52Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Pachyderm CLI and pachd server version (use pachctl version):

(.venv) ➜  tmp pachctl version
COMPONENT           VERSION
pachctl             1.12.3
pachd               1.12.3

Cloud provider (e.g. aws, azure, gke) or local deployment (e.g. minikube vs dockerized k8s):

on prem k8s

OS (e.g. from /etc/os-release):

(.venv) ➜  tmp cat /etc/os-release
NAME=Fedora
VERSION="33 (Thirty Three)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Thirty Three)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:33"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=33
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=33
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"

Your worker nodes are running out of disk space. If you look at the log you got from running kubectl logs pipeline-test-spreading-data-v1-qc4qb -c user, you'll see a number of lines like the following:
errored downloading data after 1m5.569215947s: open /pfs/.scratch/55b95bc296b04a74b4a7d0da3686db96/in/059152964-f6vMmLjNZM.json: no space left on device
With empty_files=true, I believe that even though we don't download the contents of the files, we still create metadata and references to the files locally on the workers. How much storage space is available to your worker nodes?

@dgeorg42 , I don't have access to k8s nodes but I noticed that those have ~ 20 Gb free disk space. I have a doubt it is related to not enough space because it happens right after worker is started.

Can you run kubectl describe pod pipeline-test-spreading-data-v1-qc4qb and post the results here?

 ➜ kubectl describe pod pipeline-test-spreading-data-v1-njttf
Name:         pipeline-test-spreading-data-v1-njttf
Namespace:    pachyderm
Priority:     0
Node:         somenode
Start Time:   Thu, 25 Feb 2021 21:50:09 +0200
Labels:       app=pipeline-test-spreading-data-v1
              component=worker
              istio.io/rev=default
              pipelineName=test-spreading-data
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=pipeline-test-spreading-data-v1
              service.istio.io/canonical-revision=latest
              suite=pachyderm
Annotations:
              kubernetes.io/psp: 50-restricted-istio-sidecar
              pipelineName: test-spreading-data
              seccomp.security.alpha.kubernetes.io/pod: docker/default
              sidecar.istio.io/status:
                {"version":"8e6e902b765af607513b28d284940ee1421e9a0d07698741693b2663c7161c11","initContainers":["istio-init"],"containers":["istio-proxy"]...
              specCommit: 3fdfdbcc89cd498f85f626ab33b1bfe7
              version: 1.12.3
Status:       Running
IP:           someip
IPs:
  IP:           someip
Controlled By:  ReplicationController/pipeline-test-spreading-data-v1
Init Containers:
  init:
    Container ID:  docker://5cec4f81369a5734728668fd09c272c4dc51462162667ee318fc95e621ee8d80
    Image:         docker.io/pachyderm/worker:1.12.3
    Image ID:      docker-pullable://pachyderm/worker@sha256:53bc4ebe54e3d5b4c8724b83aa4ef7c1ab96d63291e48698048da5851e9fcc8e
    Port:          <none>
    Host Port:     <none>
    Command:
      /app/init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 25 Feb 2021 21:50:11 +0200
      Finished:     Thu, 25 Feb 2021 21:50:11 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:                2
      ephemeral-storage:  10G
      memory:             4G
    Environment:          <none>
    Mounts:
      /pach-bin from pach-bin (rw)
      /pach-cache from pach-tmp (rw)
      /pfs from pachyderm-worker (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pachyderm-worker-token-xb9qq (ro)
  istio-init:
    Container ID:  docker://9b14dbd0ae82477828ccdf04dd5c1f578d48c8661264e5f84b7cbb400cf9c838
    Image:         docker.io/istio/proxyv2:1.7.3
    Image ID:      docker-pullable://istio/proxyv2@sha256:6169d096fe60f128f1311f76c97ee1c3e5d760a45042d5c9182492745d34c658
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 25 Feb 2021 21:50:12 +0200
      Finished:     Thu, 25 Feb 2021 21:50:12 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     10m
      memory:  10Mi
    Environment:
      DNS_AGENT:
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from pachyderm-worker-token-xb9qq (ro)
Containers:
  user:
    Container ID:  docker://10cf49867181ec83dc994ed1cc21c79d9fb85f38ed48bf3eb5ddbbf923a5fc81
    Image:         ds-etl-image:sometag
    Image ID:      docker-pullable://ds-etl-image@sha256:913dae511ddca5fa284cf42786199161bceb4d8c26990e3476412af5961b6aa2
    Port:          <none>
    Host Port:     <none>
    Command:
      /pach-bin/worker
    State:          Running
      Started:      Thu, 25 Feb 2021 21:50:13 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:                2
      ephemeral-storage:  10G
      memory:             4G
    Environment:
      PPS_PIPELINE_NAME:     test-spreading-data
      PACH_ROOT:             /opt/pach
      PACH_CACHE_ROOT:       /pach-cache
      PACH_NAMESPACE:        pachyderm
      PACH_IN_WORKER:        true
      PPS_WORKER_IP:          (v1:status.podIP)
      PPS_ETCD_PREFIX:       pachyderm/1.7.0/pachyderm_pps
      PPS_POD_NAME:          pipeline-test-spreading-data-v1-njttf (v1:metadata.name)
      PPS_SPEC_COMMIT:       3fdfdbcc89cd498f85f626ab33b1bfe7
      PPS_WORKER_GRPC_PORT:  1980
      PEER_PORT:             653
      STORAGE_BACKEND:       AMAZON
      GOOGLE_BUCKET:         <set to the key 'google-bucket' in secret 'pachyderm-storage-secret'>        Optional: true
      GOOGLE_CRED:           <set to the key 'google-cred' in secret 'pachyderm-storage-secret'>          Optional: true
      MICROSOFT_CONTAINER:   <set to the key 'microsoft-container' in secret 'pachyderm-storage-secret'>  Optional: true
      MICROSOFT_ID:          <set to the key 'microsoft-id' in secret 'pachyderm-storage-secret'>         Optional: true
      MICROSOFT_SECRET:      <set to the key 'microsoft-secret' in secret 'pachyderm-storage-secret'>     Optional: true
      MINIO_BUCKET:          <set to the key 'minio-bucket' in secret 'pachyderm-storage-secret'>         Optional: true
      MINIO_ENDPOINT:        <set to the key 'minio-endpoint' in secret 'pachyderm-storage-secret'>       Optional: true
      MINIO_ID:              <set to the key 'minio-id' in secret 'pachyderm-storage-secret'>             Optional: true
      MINIO_SECRET:          <set to the key 'minio-secret' in secret 'pachyderm-storage-secret'>         Optional: true
      MINIO_SECURE:          <set to the key 'minio-secure' in secret 'pachyderm-storage-secret'>         Optional: true
      MINIO_SIGNATURE:       <set to the key 'minio-signature' in secret 'pachyderm-storage-secret'>      Optional: true
      AMAZON_REGION:         <set to the key 'amazon-region' in secret 'pachyderm-storage-secret'>        Optional: true
      AMAZON_BUCKET:         <set to the key 'amazon-bucket' in secret 'pachyderm-storage-secret'>        Optional: true
      AMAZON_ID:             <set to the key 'amazon-id' in secret 'pachyderm-storage-secret'>            Optional: true
      AMAZON_SECRET:         <set to the key 'amazon-secret' in secret 'pachyderm-storage-secret'>        Optional: true
      AMAZON_TOKEN:          <set to the key 'amazon-token' in secret 'pachyderm-storage-secret'>         Optional: true
      AMAZON_VAULT_ADDR:     <set to the key 'amazon-vault-addr' in secret 'pachyderm-storage-secret'>    Optional: true
      AMAZON_VAULT_ROLE:     <set to the key 'amazon-vault-role' in secret 'pachyderm-storage-secret'>    Optional: true
      AMAZON_VAULT_TOKEN:    <set to the key 'amazon-vault-token' in secret 'pachyderm-storage-secret'>   Optional: true
      AMAZON_DISTRIBUTION:   <set to the key 'amazon-distribution' in secret 'pachyderm-storage-secret'>  Optional: true
      CUSTOM_ENDPOINT:       <set to the key 'custom-endpoint' in secret 'pachyderm-storage-secret'>      Optional: true
      RETRIES:               <set to the key 'retries' in secret 'pachyderm-storage-secret'>              Optional: true
      TIMEOUT:               <set to the key 'timeout' in secret 'pachyderm-storage-secret'>              Optional: true
      UPLOAD_ACL:            <set to the key 'upload-acl' in secret 'pachyderm-storage-secret'>           Optional: true
      REVERSE:               <set to the key 'reverse' in secret 'pachyderm-storage-secret'>              Optional: true
      PART_SIZE:             <set to the key 'part-size' in secret 'pachyderm-storage-secret'>            Optional: true
      MAX_UPLOAD_PARTS:      <set to the key 'max-upload-parts' in secret 'pachyderm-storage-secret'>     Optional: true
      DISABLE_SSL:           <set to the key 'disable-ssl' in secret 'pachyderm-storage-secret'>          Optional: true
      NO_VERIFY_SSL:         <set to the key 'no-verify-ssl' in secret 'pachyderm-storage-secret'>        Optional: true
      OBJ_LOG_OPTS:          <set to the key 'log-options' in secret 'pachyderm-storage-secret'>          Optional: true
    Mounts:
      /opt/pach from pach-dir-volume (rw)
      /pach-bin from pach-bin (rw)
      /pach-cache from pach-tmp (rw)
      /pachyderm-storage-secret from pachyderm-storage-secret (rw)
      /pfs from pachyderm-worker (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pachyderm-worker-token-xb9qq (ro)
  storage:
    Container ID:  docker://5e33a213e6435908ff9465b19e13b3f8c67044a3ad88ca5089543d78a8fec5a5
    Image:         docker.io/pachyderm/pachd:1.12.3
    Image ID:      docker-pullable://pachyderm/pachd@sha256:7f599d76e3436f9db39822ed9e77ba73586a5906dd445d43a368ca61dc533519
    Port:          <none>
    Host Port:     <none>
    Command:
      /pachd
      --mode
      sidecar
    State:          Running
      Started:      Thu, 25 Feb 2021 21:50:14 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     0
      memory:  64M
    Environment:
      PACH_ROOT:                         /opt/pach
      PACH_CACHE_ROOT:                   /pach-cache
      PACH_NAMESPACE:                    pachyderm
      BLOCK_CACHE_BYTES:                 64M
      PFS_CACHE_SIZE:                    16
      STORAGE_BACKEND:                   AMAZON
      PORT:                              650
      HTTP_PORT:                         652
      PEER_PORT:                         653
      PPS_SPEC_COMMIT:                   3fdfdbcc89cd498f85f626ab33b1bfe7
      PACHD_POD_NAME:                    pipeline-test-spreading-data-v1-njttf (v1:metadata.name)
      GC_PERCENT:                        50
      STORAGE_BACKEND:                   AMAZON
      GOOGLE_BUCKET:                     <set to the key 'google-bucket' in secret 'pachyderm-storage-secret'>        Optional: true
      GOOGLE_CRED:                       <set to the key 'google-cred' in secret 'pachyderm-storage-secret'>          Optional: true
      MICROSOFT_CONTAINER:               <set to the key 'microsoft-container' in secret 'pachyderm-storage-secret'>  Optional: true
      MICROSOFT_ID:                      <set to the key 'microsoft-id' in secret 'pachyderm-storage-secret'>         Optional: true
      MICROSOFT_SECRET:                  <set to the key 'microsoft-secret' in secret 'pachyderm-storage-secret'>     Optional: true
      MINIO_BUCKET:                      <set to the key 'minio-bucket' in secret 'pachyderm-storage-secret'>         Optional: true
      MINIO_ENDPOINT:                    <set to the key 'minio-endpoint' in secret 'pachyderm-storage-secret'>       Optional: true
      MINIO_ID:                          <set to the key 'minio-id' in secret 'pachyderm-storage-secret'>             Optional: true
      MINIO_SECRET:                      <set to the key 'minio-secret' in secret 'pachyderm-storage-secret'>         Optional: true
      MINIO_SECURE:                      <set to the key 'minio-secure' in secret 'pachyderm-storage-secret'>         Optional: true
      MINIO_SIGNATURE:                   <set to the key 'minio-signature' in secret 'pachyderm-storage-secret'>      Optional: true
      AMAZON_REGION:                     <set to the key 'amazon-region' in secret 'pachyderm-storage-secret'>        Optional: true
      AMAZON_BUCKET:                     <set to the key 'amazon-bucket' in secret 'pachyderm-storage-secret'>        Optional: true
      AMAZON_ID:                         <set to the key 'amazon-id' in secret 'pachyderm-storage-secret'>            Optional: true
      AMAZON_SECRET:                     <set to the key 'amazon-secret' in secret 'pachyderm-storage-secret'>        Optional: true
      AMAZON_TOKEN:                      <set to the key 'amazon-token' in secret 'pachyderm-storage-secret'>         Optional: true
      AMAZON_VAULT_ADDR:                 <set to the key 'amazon-vault-addr' in secret 'pachyderm-storage-secret'>    Optional: true
      AMAZON_VAULT_ROLE:                 <set to the key 'amazon-vault-role' in secret 'pachyderm-storage-secret'>    Optional: true
      AMAZON_VAULT_TOKEN:                <set to the key 'amazon-vault-token' in secret 'pachyderm-storage-secret'>   Optional: true
      AMAZON_DISTRIBUTION:               <set to the key 'amazon-distribution' in secret 'pachyderm-storage-secret'>  Optional: true
      CUSTOM_ENDPOINT:                   <set to the key 'custom-endpoint' in secret 'pachyderm-storage-secret'>      Optional: true
      RETRIES:                           <set to the key 'retries' in secret 'pachyderm-storage-secret'>              Optional: true
      TIMEOUT:                           <set to the key 'timeout' in secret 'pachyderm-storage-secret'>              Optional: true
      UPLOAD_ACL:                        <set to the key 'upload-acl' in secret 'pachyderm-storage-secret'>           Optional: true
      REVERSE:                           <set to the key 'reverse' in secret 'pachyderm-storage-secret'>              Optional: true
      PART_SIZE:                         <set to the key 'part-size' in secret 'pachyderm-storage-secret'>            Optional: true
      MAX_UPLOAD_PARTS:                  <set to the key 'max-upload-parts' in secret 'pachyderm-storage-secret'>     Optional: true
      DISABLE_SSL:                       <set to the key 'disable-ssl' in secret 'pachyderm-storage-secret'>          Optional: true
      NO_VERIFY_SSL:                     <set to the key 'no-verify-ssl' in secret 'pachyderm-storage-secret'>        Optional: true
      OBJ_LOG_OPTS:                      <set to the key 'log-options' in secret 'pachyderm-storage-secret'>          Optional: true
      STORAGE_UPLOAD_CONCURRENCY_LIMIT:  100
    Mounts:
      /opt/pach from pach-dir-volume (rw)
      /pachyderm-storage-secret from pachyderm-storage-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pachyderm-worker-token-xb9qq (ro)
  istio-proxy:
    Container ID:  docker://122ecb85ef0c149095376b5414977ec0a058a5c6c7fc3f610db3e16f7a422b8c
    Image:         docker.io/istio/proxyv2:1.7.3
    Image ID:      docker-pullable://istio/proxyv2@sha256:6169d096fe60f128f1311f76c97ee1c3e5d760a45042d5c9182492745d34c658
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      pipeline-test-spreading-data-v1.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --trust-domain=cluster.local
      --log_as_json
      --concurrency
      2
    State:          Running
      Started:      Thu, 25 Feb 2021 21:50:14 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      10m
      memory:   40Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    first-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      pipeline-test-spreading-data-v1-njttf (v1:metadata.name)
      POD_NAMESPACE:                 pachyderm (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      CANONICAL_SERVICE:              (v1:metadata.labels['service.istio.io/canonical-name'])
      CANONICAL_REVISION:             (v1:metadata.labels['service.istio.io/canonical-revision'])
      PROXY_CONFIG:                  {"tracing":{"zipkin":{"address":"tracing-collector.istio-system:9411"}},"proxyMetadata":{"DNS_AGENT":""}}

      ISTIO_META_POD_PORTS:          [
                                     ]
      ISTIO_META_APP_CONTAINERS:     user,storage
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_METAJSON_ANNOTATIONS:    {"authTokenHash":"somehas","kubernetes.io/psp":"50-restricted-istio-sidecar","pipelineName":"test-spreading-data","seccomp.security.alpha.kubernetes.io/pod":"docker/default","specCommit":"3fdfdbcc89cd498f85f626ab33b1bfe7","version":"1.12.3"}

      ISTIO_META_WORKLOAD_NAME:      pipeline-test-spreading-data-v1
      ISTIO_META_OWNER:              kubernetes://apis/v1/namespaces/pachyderm/replicationcontrollers/pipeline-test-spreading-data-v1
      ISTIO_META_MESH_ID:            cluster.local
      DNS_AGENT:
      ISTIO_KUBE_APP_PROBERS:        {}
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pachyderm-worker-token-xb9qq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  pach-bin:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  pach-tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  pachyderm-worker:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  pach-dir-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  pachyderm-storage-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pachyderm-storage-secret
    Optional:    false
  pachyderm-worker-token-xb9qq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pachyderm-worker-token-xb9qq
    Optional:    false
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istiod-ca-cert:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        istio-ca-root-cert
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From                  Message
  ----    ------     ----       ----                  -------
  Normal  Scheduled  <unknown>  default-scheduler     Successfully assigned pachyderm/pipeline-test-spreading-data-v1-njttf to somenode
  Normal  Pulled     18m        kubelet, somenode  Container image "docker.io/pachyderm/worker:1.12.3" already present on machine
  Normal  Created    18m        kubelet, somenode  Created container init
  Normal  Started    18m        kubelet, somenode  Started container init
  Normal  Pulled     17m        kubelet, somenode  Container image "docker.io/istio/proxyv2:1.7.3" already present on machine
  Normal  Created    17m        kubelet, somenode  Created container istio-init
  Normal  Started    17m        kubelet, somenode  Started container istio-init
  Normal  Created    17m        kubelet, somenode  Created container user
  Normal  Pulled     17m        kubelet, somenode  Container image "ds-etl-image:sometag" already present on machine
  Normal  Started    17m        kubelet, somenode  Started container user
  Normal  Pulled     17m        kubelet, somenode  Container image "docker.io/pachyderm/pachd:1.12.3" already present on machine
  Normal  Created    17m        kubelet, somenode  Created container storage
  Normal  Started    17m        kubelet, somenode  Started container storage
  Normal  Pulled     17m        kubelet, somenode  Container image "docker.io/istio/proxyv2:1.7.3" already present on machine
  Normal  Created    17m        kubelet, somenode  Created container istio-proxy
  Normal  Started    17m        kubelet, somenode  Started container istio-proxy

I think what's likely happening here is that you're running out of inodes rather than actual space on the device. I think the only way around this is to rearchitect the pipeline in a way that it doesn't require creating millions of files. does your code need to see all of the files at the same time?

essentially, I just wanted to trigger some pipeline when there were changes in preceding pipeline. Since preceeding pipeline produced a lot of data, I tried to avoid downloading such data by using empty_files option.

I see, so don't care about the actual data, you just want to trigger something when the data is available. That's a little bit awkward with the current pipeline semantics, they're tuned around things being data driven so the data is implicitly included. A few ideas that come to mind:

you can use a drop file and change your glob to /drop-file instead of /. The tricky part of this is that you have to generate the drop file and make the contents change when you want to trigger. Otherwise the computation gets deduplicated.
You can use the underyling APIs to subscribe to commits and run your code when you get one. This has the downside that you need to write your own service rather than having Pachyderm manage it, but you can do whatever you want with it.
Similarly you can have the code that commits data call FlushCommit to wait for processing to finish and then do what you want with the results.

Hopefully one of those works for you or at least gives you a direction. If not feel free to swing by our Slack channel and we'll be happy to help with architecting your pipeline. If none of that works we might have to add a new feature to facilitate this, it shouldn't be too big of a lift.

thanks


Add and test encapsulation

I don't know if we can upload this to pypi yet, because the Lean 4 based executable might not be able to run in a standalone environment.

I think we have two main ways to handle this:

Pre-building and uploading artifacts to PyPI
This could work if the builds aren't too large (PyPI limits: single file size to 100 MiB and total project size to 10 GiB). It's the most user-friendly approach - just pip install and go.

Building during install or runtime
This is more flexible. We could either build on first run, or just provide build scripts and let users handle it.

I'd lean towards pre-building - it's simpler for everyone. For the dynamic building approach, we could leverage GitHub Actions to automate the build process and provide better cross-platform support.

Regarding your comment about LeanDojo in the article:

Pantograph has several key architectural improvements over LeanDojo. First of all, it is written entirely in Lean 4. This removes the need for external dependencies (such as Docker) and also improves the speed of interaction. Also, some tactics not supported by LeanDojo, such as have, are available in Pantograph. Other tactics, such as conv and calc...

LeanDojo has actually been Docker-free for quite some time. It interact with traced repositories, relying on two Lean scripts (Lean4Repl.lean and ExtractData.lean). The Python package serves merely as a wrapper, similar to PyPantograph.
The tactic support (such as 'have' and 'calc') is indeed a limitation in LeanDojo, and this makes Pantograph more valuable. :)

Can the pre-built Pantograph repl run in an environment with no lake or lean?

I believe not. Perhaps we could adopt the second method - building in .pantograph when the user first uses Pantograph and no cache is found.
Currently, the repository uses git submodules to fetch Lean code, which might take some effort to migrate to PyPI.
For immediate development needs, I suggest:

Focus on fixing these installation bugs
Set up a CI pipeline for testing in the current setting
When the development is ready for release, consider a different approach:

Separate the development of lean repl and the python wrapper.
Cache builds in $HOME/.pantograph, similar to .elan

This approach could also enable support for multiple versions based on project-specific lean-toolchain requirements.

BTW, I get an installation issue when using Conda.
One needs to use pip install -e . or manually copy the pantograph-repl to the environment:
cp PyPantograph/pantograph/pantograph-repl $HOME/miniconda/envs/pantograph/lib/python3.11/site-packages/pantograph

Otherwise, the Python code only works within the PyPantograph folder.
This issue would be resolved with the caching approach.

The interaction bugs are fixed and I'm waiting for user feedback to close the issues and merge the PRs into mainline (deadline by the end of this week).
The Lean repl is already in a separate repository, so its separate.
I don't use conda, so I will not be able to help you with conda issues. I think you'll  have to work with pip install -e ..
I'll do the housekeeping work after we figure out a stable way to deliver this software to the users.


Update build-unix.md

Replaced Core with Classic

We don't accept patches into master at this time, all the work is done in branch 0.11.2.


libhugetlbfs: mark supported on aarch64

Description of changes
Was previously broken, but now builds.
Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg build libhugetlbf

https://github.com/ofborg build libhugetlbfs

@ofborg build libhugetlbfs


improve the debug interface a bit and update the error message

this should be quick and easy. are const fns the way to go or should we prefer to use associated constants?
changelog:
- Added `impl BitOr` for `debug::MessageType` and `debug::MessageSeverity` and some bit field generators, also turning them into `const fn`.

@incertia Merged. Thank you for the improvement!


Can't find package.json

I've uploaded an archive.zip containing a file called 'main.js' and another called 'package.json'. The checker tells me I fail every test.
Here is the error:
internal/fs/utils.js:230
throw err;
^
Error: ENOENT: no such file or directory, open 'package.json'
at Object.openSync (fs.js:457:3)
at Object.readFileSync (fs.js:359:35)
at Object. (/home/alf/main.js:7:19)
at Module._compile (internal/modules/cjs/loader.js:1158:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
at Module.load (internal/modules/cjs/loader.js:1002:32)
at Function.Module._load (internal/modules/cjs/loader.js:901:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
at internal/main/run_main_module.js:18:47 {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: 'package.json'
}
internal/fs/utils.js:230
throw err;
^
And here I attach the contents of the .json file:
{
"author": "Dragos",
"version": "1.0.0",
"main": "main.js",
"dependencies": {
"cowsay": "^1.4.0"
}
}

You can try to update your node.js to the latest version, maybe that works. Did a little research online.

I updated node.js using chocolatey. I ran the program once more and uploaded it again. The result is the same.

I also tried creating another package.json. Still no success.

We are using node v12, upload your homework to vmchecker and if you have the same error, please post a screenshot here.

On 1 Mar 2020, at 17:56, dragossolca<EMAIL_ADDRESS>wrote:

I also tried creating another package.json. Still no success.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

The checker runs on version 12 of Node.js trying to downgrade and after trying again.

what is your username?

Maybe because you need 4 files on your .zip, and you have only 3 : Readme.txt , package.json and main.js . There should be another one : package-lock.json . I think you didn't select it when you archived the files.
I created a new main and a new package.json and ran them with node 12(the lts one).
I also added the package-lock.json in the archive.
Still ENOENT

You are trying to read the file package.json form the current folder. When
testing your homework, the file is not there, as the current path is
different. When opening the file, use __dirname+'/package.json'.
On Mon, Mar 2, 2020 at 6:06 PM dragossolca<EMAIL_ADDRESS>wrote:

dragos.solca is my username.
I created a new main and a new package.json and ran them with node 12(the
lts one).
I also added the package-lock.json in the archive.
Still ENOENT
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/UPB-FILS/alf/issues/8?email_source=notifications&email_token=AAVU5IPXHWZESIMT6MMW5FDRFPKRPA5CNFSM4K7EQESKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENP4GWI#issuecomment-593478489,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAVU5IPXIKG3KARFCN62NUTRFPKRPANCNFSM4K7EQESA
.

--
Alexandru RADOVICI
Universitatea "Politehnica" din Bucuresti
e-mail<EMAIL_ADDRESS>telefon:<PHONE_NUMBER>
www.ipworkshop.ro

I' m truly grateful. Thank you for your help. It works.


Add GitHub Issue Template Schema Form

Description:
To streamline the process of creating issues and ensure consistency, we need to add a GitHub issue template schema form. This will help contributors easily provide all the necessary details when reporting bugs, suggesting features, or requesting documentation.
Requirements:

Create an issue template that includes fields for:

Issue Type (Bug, Feature Request,Documentation)
Description
Steps to Reproduce (for Bug reports)
Expected Outcome
Additional Context or Screenshots

Ensure the template is easy to navigate and fills out the required fields.

Proposed Changes:

Implement the issue template in .github/ISSUE_TEMPLATE folder.
Test the template to verify the form fields populate correctly.

Before and After:
Before:

After:

Additional Information:
If you have any suggestions for extra fields or modifications, please mention them in the comments.
PS: I am willing to submit the Pull Request

Please do! This would be a welcome change

Gotcha! Can you assign it to me?

@creatorrr, Proposed the Pull Request!


[Feat] Support  viewing output for watch mode

Most test framework has watch mode, that watch src file, auto compile and run test.
But it's likely neotest wait until test process finish before fullfill the test result.
The consequence is, even when all test cases are finished in watch  mode,   the test output is not fullfilled into neotest.
Hence, here is the result of neotest output

Can we provide the mechanism to update the test output in these cases?

You can attach to the running process to see the output, using either the attach mapping in the summary or neotest.run.attach()

You can attach to the running process to see the output, using either the attach mapping in the summary or neotest.run.attach()

Thanks, it resolves my issue perfectly.


chore: use different keyring for differney ci run

Close this PR as we agreed we will use 1 keyring with multiple keys in it.


Auto Snapshot Releasing

Add a workflow action to release alphas of packages if they have changes to be release each time A PR is merged into main and then attempt to install that package.
for testing that packages can be published and installed without issue

This is NOT working. Do not merge in


atKeys being erased for no good reasons ??

Describe the bug
Using sshnoports an atKey is set for the username this is set with a 0 TTL as can be seen in the Metadata
PS C:\Users\colin\GitHub\@foundation\sshnoports> dart run bin/sshnp.dart -f "@iot_manager" -t "@iot_device01" -d iot_device01 --host <IP_ADDRESS> -s GitHub_rsa.pub
>>AtValue{value: cconstab, metadata: Metadata{ttl: 0, ttb: 0, ttr: null,ccd: false, isPublic: false, isHidden: false, availableAt : 2023-03-29 22:59:40.406Z, expiresAt : null, refreshAt : null, createdAt : 2023-02-22 06:17:55.612Z, updatedAt : 2023-03-29 22:59:40.406Z, isBinary : false, isEncrypted : true, isCached : false, dataSignature: null, sharedKeyStatus: null, encryptedSharedKey: cTB3PvWFpnivMA/7dRe4DlzM/LwWbFeKYfJEOtQc2oxHsO7m2ChqougN/JOKAM6iDAGNcAgspxCKI7slXwssETB7qEjqxoNCR3xD65Jm/2Oex2vHWaj+NgKzzNbESGf5UApl2icNDHjiqymCDTy3rxaWK9BJL8FgXc7DL0wRHbIhbQeOSJlg1jOLoNV3DXdpyNA8pbnq8GhAjSao+vTmtifItxuCC9meqnO5SUFcceBiTnR7MdZy7FhrVgbcSf45AQW/18/twQ2iM/dshBHiZHsbZ4c2iV/WckmmC3M4T9VJyNDlttIOxG2GoIGGy5NDnVcZEny9Ec0yeZvJbsDjDg==, pubKeyCheckSum: a8f33d7479676fb1d9d760ac40f1a0ea, encoding: null, encKeyName: null, encAlgo: null, ivNonce: KZd9BIfbSvE/R4K6AqIlmg==, skeEncKeyName: null, skeEncAlgo: null}}
Remote sshnpd error: Remote SSH Client failure : SSHAuthFailError(All authentication methods failed)

but after a few minutes - 20 to 30 the atKey goes away. There is no code to do this in the sshnp client so looks to me like a serious bug in the SDK/atServer ?
PS C:\Users\colin\GitHub\@foundation\sshnoports> dart run bin/sshnp.dart -f "@iot_manager" -t "@iot_device01" -d iot_device01 --host <IP_ADDRESS> -s GitHub_rsa.pub
>>null
Device "iot_device01" unknown or username not shared
PS C:\Users\colin\GitHub\@foundation\sshnoports>

Steps to reproduce

set an atKey
Then share it with another atSign
And then look it up until is returns a null
where did it go ?

Expected behavior
The atKey should not disappear it never did in the past !
Screenshots
How the atKey is set
  if (results['username']) {
    var metaData = Metadata()
      ..isPublic = false
      ..isEncrypted = true
      ..namespaceAware = true;

    var atKey = AtKey()
      ..key = "username.$device"
      ..sharedBy = deviceAtsign
      ..sharedWith = managerAtsign
      ..namespace = nameSpace
      ..metadata = metaData;

    await atClient.put(atKey, username);
  }

how the atKey is read
  var metaData = Metadata()
    ..isPublic = false
    ..isEncrypted = true
    ..namespaceAware = true;

  var atKey = AtKey()
    ..key = "username"
    ..sharedBy = toAtsign
    ..sharedWith = fromAtsign
    ..namespace = nameSpace
    ..metadata = metaData;
  AtValue? toAtsignUsername;
  try {
    toAtsignUsername = await atClient.get(atKey);
    print('>>'+toAtsignUsername.toString());
  } catch (e) {
      print('>>'+toAtsignUsername.toString());
    stderr.writeln("Device \"${device.replaceAll('.', '')}\" unknown or username not shared");
    await cleanUp(sessionId, logger);
    exit(1);
  }

Smartphones
latest SDK/Dart
Windows/Linux/Mac
Were you using an atApplication when the bug was found?
sshnp v3.0.0
Additional context
No response

included some additional code so you can see the atKey in the error

  var metaData = Metadata()
    ..isPublic = false
    ..isEncrypted = true
    ..namespaceAware = true;

  var atKey = AtKey()
    ..key = "username"
    ..sharedBy = toAtsign
    ..sharedWith = fromAtsign
    ..namespace = nameSpace
    ..metadata = metaData;
  AtValue? toAtsignUsername;
  try {
    toAtsignUsername = await atClient.get(atKey);
    print('>> $atKey');
    print('>>' + toAtsignUsername.toString());
  } catch (e) {
    print('>> $atKey');
    print('>>' + toAtsignUsername.toString());
    stderr.writeln("Device \"${device.replaceAll('.', '')}\" unknown or username not shared");
    await cleanUp(sessionId, logger);
    exit(1);
  }

When the atKey has been deleted (assuming deleted) after 20-30 mins (cron job ??)
PS C:\Users\colin\GitHub\@foundation\sshnoports> dart run bin/sshnp.dart -f "@iot_manager" -t "@iot_device01" -d iot_device01 --host <IP_ADDRESS> -s GitHub_rsa.pub
>> @iot_manager:username.iot_device01.sshnp@iot_device01
>>null
Device "iot_device01" unknown or username not shared
PS C:\Users\colin\GitHub\@foundation\sshnoports>

and once  I restart trhe sshnpd so the atKey is reshared
PS C:\Users\colin\GitHub\@foundation\sshnoports> dart run bin/sshnp.dart -f "@iot_manager" -t "@iot_device01" -d iot_device01 --host <IP_ADDRESS> -s GitHub_rsa.pub
>> @iot_manager:username.iot_device01.sshnp@iot_device01
>>AtValue{value: cconstab, metadata: Metadata{ttl: 0, ttb: 0, ttr: null,ccd: false, isPublic: false, isHidden: false, availableAt : 2023-03-30 02:26:50.412Z, expiresAt : null, refreshAt : null, createdAt : 2023-02-22 06:17:55.612Z, updatedAt : 2023-03-30 02:26:50.412Z, isBinary : false, isEncrypted : true, isCached : false, dataSignature: null, sharedKeyStatus: null, encryptedSharedKey: e34hWPoek5BAPXhKugQMT8fbCVzRsPckzsNWWzr7PnWcOe2OnHwMh956juE07Gsyht5wW2gqR9d7aRLrXvxXT7q5G1esaSgj95Z5OPyLVXphdEAwIHG6BhFed/nR9kNVTk8yyIbE/8pwIwuAGp9yv3bDA8BQPCn0Sf1Iya92e/SpDTnjUfkxvR3AgNOrOczi6F0PWQzmEuN/9ZHDXop9FWS0Cgu3y5aVT1Ev8hI3Z/h3cYMjWT+ugkzcuHyJhNtP/VhCbCKuVgmUj1uFCtRUUAdoSRxgzcfRVKOAEyaeMyZrYs8szmdR1LRlczaTISgDbYIIVZtAKW4upHaL5n6Ygw==, pubKeyCheckSum: a8f33d7479676fb1d9d760ac40f1a0ea, encoding: null, encKeyName: null, encAlgo: null, ivNonce: yEOWe5U/mwbXmk9PCf1tdg==, skeEncKeyName: null, skeEncAlgo: null}}
Remote sshnpd error: Remote SSH Client failure : SSHAuthFailError(All authentication methods failed)

PS C:\Users\colin\GitHub\@foundation\sshnoports>

Old client does not understand the metadata so we loose the IV in the metadata

Incident over. We have work to do with regards to consistently and systematically applying a patch semantic to updates to ensure that metadata is preserved. In this case, an old client, unaware of new metadata, sent updates to the server which didn't include the new metadata. The server should have preserved the metadata (in this case the ivNonce) - i.e. we need Patch rather than Put semantics. (These are http terms but they apply here also. https://www.geeksforgeeks.org/difference-between-put-and-patch-request/ )

Closing this ticket as we've created two new tickets for the changes that need to be made


Crash occurring when trying to force unwrap possible nil response header

Describe the bug
A crash is occurring when trying to verify the users email with a verification code with amplify.
The toString() function below in Utilities.swift crashes due to possible force unwrapping when nil
extension aws_byte_cursor {
    func toString() -> String {
        if self.len == 0 {
            return ""
        }

        let data = Data(bytesNoCopy: self.ptr, count: self.len, deallocator: .none)
        return String(data: data, encoding: .utf8)!
    }

    func toOptionalString() -> String? {
        if self.len == 0 { return nil }
        let data = Data(bytesNoCopy: self.ptr, count: self.len, deallocator: .none)
        return String(data: data, encoding: .utf8)!
    }
}

Expected Behavior
No crash
Current Behavior
Thread 8 Crashed:
0   App                             0x101570abc         [inlined] value
1   App                             0x101570abc         [inlined] aws_byte_cursor.toString (Utilities.swift:160)
2   App                             0x101570abc         [inlined] HTTPHeader.init (HTTPHeader.swift:18)
3   App                             0x101570abc         [inlined] HTTPHeader.__allocating_init (HTTPHeader.swift:19)
4   App                             0x101570abc         [inlined] onResponseHeaders (HTTPStreamCallbackCore.swift:56)
5   App                             0x101570abc         [inlined] Sequence.forEach
6   App                             0x101570abc         onResponseHeaders (HTTPStreamCallbackCore.swift:56)
7   App                             0x101570960         [inlined] aws_byte_cursor.toString (Utilities.swift:160)
8   App                             0x101570960         [inlined] HTTPHeader.init (HTTPHeader.swift:18)
9   App                             0x101570960         [inlined] HTTPHeader.__allocating_init (HTTPHeader.swift:19)
10  App                             0x101570960         [inlined] onResponseHeaders (HTTPStreamCallbackCore.swift:56)
11  App                             0x101570960         [inlined] Sequence.forEach
12  App                             0x101570960         onResponseHeaders (HTTPStreamCallbackCore.swift:56)
13  App                             0x1015bb048         s_decoder_on_header (h1_connection.c:1156)
14  App                             0x1015bc8a4         s_linestate_header (h1_decoder.c:534)
15  App                             0x1015bc1f0         s_state_getline (h1_decoder.c:126)
16  App                             0x1015bb9a4         aws_h1_decode (h1_decoder.c:724)
17  App                             0x1015b91e4         [inlined] s_try_process_next_stream_read_message (h1_connection.c:1863)
18  App                             0x1015b91e4         aws_h1_connection_try_process_read_messages (h1_connection.c:1755)
19  App                             0x1015ba31c         s_handler_process_read_message (h1_connection.c:1721)
20  App                             0x1015a2a04         s_process_read_message (secure_transport_tls_channel_handler.c:661)
21  App                             0x1015b11fc         s_do_read (socket_channel_handler.c:164)
22  App                             0x1015b0a88         s_on_readable_notification (socket_channel_handler.c:221)
23  App                             0x1015af7ec         s_on_socket_io_event (socket.c:1697)
24  App                             0x10159c744         aws_event_loop_thread (kqueue_event_loop.c:932)
25  App                             0x1015917bc         thread_fn (thread.c:177)
26  libsystem_pthread.dylib         0x40381a4d0         _pthread_start

Reproduction Steps
I personally have not been able to replicate this issue, but has been occurring with our users according to our logs.
Possible Solution
Safely unwrap possible nil response headers, e.g. use a ternary operator to return an empty string
Additional Information/Context
No response
aws-crt-swift version used
0.17.0 via amplify Swift
Compiler and Version used
XCode 15.0, Swift 5
Operating System and version
MacOS 13.5

Hi,
Thank you for reporting the issue. We are looking into it. Could you please specify the service to which the failing requests were made? Thank you.

We are using amplify-swift.
Specifically calling confirmSignUp followed by signIn

@Jeffrey-Chau-Leo, thank you for the update. I asked because this can only fail when the headers contain non-ASCII characters. Since there is no standardization for which encoding to use for headers, we are discussing multiple approaches.

@Jeffrey-Chau-Leo Do you know of any cases where the headers for SignUp or SignIn might have non-ASCII characters, perhaps as part of a username or something similar? We are thinking about limiting it to ASCII characters only and ignoring the rest.

@waahm7
Unfortunately, yes, we have users who have non-ASCII characters in their email addresses.

This is fixed in https://github.com/awslabs/aws-crt-swift/releases/tag/0.23.0. We have made the following improvements:

Fixed some bugs in our encoding/decoding logic around UTF-8. UTF-8 should now work properly.
For encodings other than UTF-8, we now try to decode using UTF-8 instead of force decoding and replace invalid bytes with the missing byte symbol (�).


blufi@nimble:  ble_gatts_notify_custom()  undefined (IDFGH-10611)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

esp-idf V4.4.5
ESP32
Issue
the output of  idf.py build:

esp-idf/components/bt/common/btc/profile/esp/blufi/nimble_host/esp_blufi.c:435:10: error: implicit declaration of function 'ble_gatts_notify_custom'; did you mean 'ble_gattc_notify_custom'? [-Werror=implicit-function-declaration]
     rc = ble_gatts_notify_custom(blufi_env.conn_id, gatt_values[1].val_handle, om);
          ^~~~~~~~~~~~~~~~~~~~~~~
          ble_gattc_notify_custom

Suggest :
should  use  ble_gattc_notify_custom()

Reference
https://mynewt.apache.org/latest/network/ble_hs/ble_gatts.html?highlight=ble_gattc_notify_custom#c.ble_gattc_notify_custom

int ble_gattc_notify_custom()
Deprecated.
Should not be used. Use ble_gatts_notify_custom instead.

Thanks. Change is already committed to github here.


Update create_pr.md

Proposed changes

Unreleased project version (optional)

Related issues (optional)

Thanks, @scjane :)


Add new adapter ioBroker.rmb-bhkw to latest repository

RE-CHECK!

Automated adapter checker
No changed adapters found
Add comment "RE-CHECK!" to start check anew

Meeh, ok the resorted adapters break the detection :-)
Results from a manual check at https://adapter-check.iobroker.in/

[E125] External icon not found under URL: https://raw.githubusercontent.com/satchafunkilus/ioBroker.rmb-bhkw/main/admin/rmb-bhkw.png
[E201] Bluefox was not found in the collaborators on NPM!. Please execute in adapter directory: "npm owner add bluefox iobroker.rmb-bhkw"

@Apollon77

Regarding the external icon: I have followed the folder structure by the adapter creation script and the external icon is located under ioBroker.rmb-bhkw/admin/rmb-bhkw.png. Is there any specific reason why this should be in an additional "main" subfolder?
I have added Bluefox as owner, already some days ago. It is showing as invitation pending in npm.

Is there any specific reason why this should be in an additional "main" subfolder?

The URLs of the Github "raw content" includes the branch!!

@Apollon77 ok, since the branch is called master and not main, the icon is located under: https://raw.githubusercontent.com/satchafunkilus/ioBroker.rmb-bhkw/master/admin/rmb-bhkw.png. Do I need to rename the branch to main?

No, then lets consider it as false positive

No, then lets consider it as false positive

Ok, understood. Is there any other to-do then on my end?

No, I will do a review soon

@Apollon77 I'll try a review here.

Hi,
here some review comments:

Must: An unmodified schedule adapter that queries data from external systems will produce a lot of issues because usually the default schedule stays unchanged for all users and so all installations out there will then contact the server at the exact same time create a "potentially huge" request spike. The best practice for now is at least to add a random delay to spread the calls within that minute - or even move the "minutes" on first start and so persist that (e.g. see https://github.com/ioBroker/ioBroker.yr/blob/master/main.js#L84-L90) ... if the adapter gets successfull then we ahve an big issue and the webpage owner contacting us ... we should prevent that from the beginning.
Hint: You can add the required apt-packages in io-package like https://github.com/ioBroker/ioBroker.phantomjs/blob/master/io-package.json#L124-L128 and iobroker will try to install them on install of the adapter
Please remove the onStateChange handler because not needed
You should await https://github.com/satchafunkilus/ioBroker.rmb-bhkw/blob/master/main.js#L154, else you might stop too soon!
I think https://github.com/satchafunkilus/ioBroker.rmb-bhkw/blob/master/main.js#L127 would pot. set a state without the object being initialized at least once?, right?

PS: regarding the main vs master ... https://github.com/ioBroker/ioBroker.repositories/pull/1779/files#diff-1384831e45cb5b5fc71b065c642c35169d0ee196221e07b4b81075a0d3b0aed8R1627 here you have it mixed up

@satchafunkilus
Hi,
some things I noticed in addition to @Apollon77:

[ ] the icon and readme paths in your io-package.json are wrong and point to main-branch. Adjust them to master branch (might be connected to your PR here, see earlier comment).
[ ] Your README is nice, but completely in German. Please add at least a short description in English (even if the adapter should only be of interest in Germany, people all over the world still need to decide if they need this adapter or not).

@Apollon77 @Garfonso thanks for your remarks, I have incorporated all your remarks and suggestions.
Regarding the schedule randomization: It appears to me that the approach suggested only works for randomizing an hourly poll of the data. As I need to poll the data every 5-15min, I added a random delay between 0-59 seconds, so not all polls are executed at the same time. I would assume that, taking into account the small number of devices of this type out there, this should be enough to prevent any major spikes on the servers.

Also a schedule like 1,6,11,16,... * * ... is a valid one ;-) but ok. I think it is still better then nothing, so ok for me ;-). Will check in depth later

Hi, you need to add one thi g if you now use such a delay option: You need to add a flag to your unload handler to remember that unload was called while delay was running ... and if then you need to stop your logic after the delay ... or you need to add a way that allows to cancel the delay in case of an unload.
500ms after your unload callback the database is gone and running your logic will give issues.
And please remove the german version from readme because readme get automatically translated and this will not work. You can add locatlized reme pages in (docs/de and docs/en) like https://github.com/iobroker-community-adapters/ioBroker.pvforecast/tree/main/docs and add it to io-package https://github.com/iobroker-community-adapters/ioBroker.pvforecast/blob/main/io-package.json#L119-L122
After this we should be ready

@satchafunkilus DO you need help for anything?

@satchafunkilus DO you need help for anything?

Thanks, I'm currently on a longer vacation without my PC, so unfortunately I can't work on the code at the moment. I tried implementing the thing you mentioned wiith regards to the flag for the delay before I left, but I wasn't really able to get it working in the short time. If you have any example for such a flag, that would help a lot. Thanks.

Sure, just set a variable :-)) e.g. https://github.com/Apollon77/ioBroker.meross/blob/master/main.js#L169
and then check in the other places "if (stopped) return" to simply stop logic :-)

@satchafunkilus Any update?

@satchafunkilus Any update?

I have added the check as you mentioned - however, I'm not 100% sure it's correct this way.  Could you have a quick look?
Thanks

RE-CHECK!

Automated adapter checker
ioBroker.rmb-bhkw

:thumbsup: No errors found

[ ] :eyes: [W171] "common.title" is deprecated in io-package.json
[ ] :eyes: [W170] "common.keywords" should not contain "iobroker, adapter, smart home" io-package.json
[ ] :eyes: [W156] Adapter should support admin 5 UI (jsonConfig)
[ ] :eyes: [W145] Each "common.news" should be translated into all supported languages (en, de, ru, pt, nl, fr, it, es, pl, zh-cn)
[ ] :eyes: [W400] Cannot find "rmb-bhkw" in latest repository

Add comment "RE-CHECK!" to start check anew

@satchafunkilus near to correct :-)  When stpped is set true now thenstop prpcess is already in progress. replace https://github.com/satchafunkilus/ioBroker.rmb-bhkw/blob/master/main.js#L54 by just a "return;". SP just make sure to not continue ...
And BTW: maybe fill in better tags? https://github.com/satchafunkilus/ioBroker.rmb-bhkw/blob/master/io-package.json#L76
Last but not least: Please also check the comments of the automatic checker above

@satchafunkilus

@Apollon77 I have finally found some time to make the changes requested above. I have tested it, and it seems to be working fine. For some reason, the automated checks on github are now failing, though – the error message is:
pm ERR! Cannot read property '@iobroker/adapter-core' of undefined
I have tried deleting all the npm packages and reinstalling them, but that didn't change anything. Any clue what I can do here?

RE-CHECK!

Automated adapter checker
ioBroker.rmb-bhkw

[ ] :heavy_exclamation_mark: [E118] Versions in package.json and in io-package.json are different
[ ] :eyes: [W202] Version of package.json (1.0.5) doesn't match latest version on NPM (1.0.4)
[ ] :eyes: [W400] Cannot find "rmb-bhkw" in latest repository

Add comment "RE-CHECK!" to start check anew

Ideally delete node-modules and also the package-lock.json and then execute "npm install". How is it then?

That I already tried yesterday, unfortunately it didn't help. But I realized that the node-modules folder is not uploaded to github anyway, so that cannot really be the issue.

Did you really also deleted the package-lock.json???? Because this one IS uploaded

Yes, both

https://t.me/ioBroker_development/26673 could be related to this.
Question:
Which npm version do you have installed?
Where exactly did you get the npm error? During tests?

Yes, that sounds like the same issue. I get it during the github actions tests after uploading a new version. https://github.com/satchafunkilus/ioBroker.rmb-bhkw/actions/runs/4213085290
npm version is 9.5.0

Please remove the tests for Node 12, which are no longer supported as of ioBroker/Testing version 4.0.0.
Your version in package.json is 4.1.0

@satchafunkilus
I made you a PR where I updated some files and took out the test for node 12 / 14.
Check if everything still works for you.

@satchafunkilus
I made you a PR where I updated some files and took out the test for node 12 / 14.
Check if everything still works for you.

Awesome, thanks. Much appreciated. Tests seem to have run through now without any issues.

RE-CHECK!

Automated adapter checker
ioBroker.rmb-bhkw

:thumbsup: No errors found

[ ] :eyes: [W400] Cannot find "rmb-bhkw" in latest repository

Add comment "RE-CHECK!" to start check anew

Any to-do left here from my end?

This PR MUST NOT try to update package-lock.json.
A PR requesting addition to latest must update exactly sources-dist.json only.
Please remove package-lock.jsosn from the PR or close this PR and create a new one. Feel free to use iobroker.dev support to create the PR with one or two clicks.

Please remove package-lock.jsosn from the PR or close this PR and create a new one. Feel free to use iobroker.dev support to create the PR with one or two clicks. When creating a ne PR, please reference this PR in it.

I have created a new PR https://github.com/ioBroker/ioBroker.repositories/pull/2298


Run formatter

Motivation
The formatter version was bumped in https://github.com/dfinity/gix-components/pull/380, changing the formatting rules.
But the formatter wasn't run.
Changes
npm run format
Screenshots
N/A

@dskloetd I don't see any file changes...


"#include "binding_callback_thread_observer.hpp"" file not found

Hi,
I saw another thread where this was discussed and followed your instructions to do:
pod cache clean Realm
pod cache clean RealmSwift
pod deintegrate || rm -rf Pods
pod install --verbose
rm -rf ~/Library/Developer/Xcode/DerivedData

Here's the output for Realm (I didn't include output for other pods):
Installing Realm (2.5.1)
 > Git download
 > Git download
     $ /usr/local/git/bin/git clone https://github.com/realm/realm-cocoa.git
     /var/folders/zv/lcmm9hx57s9crpjf3_703vnw0000gn/T/d20170424-4451-1psq2x6
     --template= --single-branch --depth 1 --branch v2.5.1
     Cloning into '/var/folders/zv/lcmm9hx57s9crpjf3_703vnw0000gn/T/d20170424-4451-1psq2x6'...
     Note: checking out 'f8d9bca087d3e4e9a13eb5f0d9bdb50f8de9eeab'.

     You are in 'detached HEAD' state. You can look around, make experimental
     changes and commit them, and you can discard any commits you make in this
     state without impacting any branches by performing another checkout.

     If you want to create a new branch to retain commits you create, you may
     do so (now or later) by using -b with the checkout command again. Example:

       git checkout -b new_branch_name

     $ /usr/local/git/bin/git -C
     /var/folders/zv/lcmm9hx57s9crpjf3_703vnw0000gn/T/d20170424-4451-1psq2x6
     submodule update --init --recursive
     Submodule 'Realm/ObjectStore' (https://github.com/realm/realm-object-store.git) registered for path 'Realm/ObjectStore'
     Cloning into 'Realm/ObjectStore'...
     Submodule path 'Realm/ObjectStore': checked out 'd88499cd5cc4e9a19a91b964d0d282da0fa51f98'
     Submodule 'external/catch' (https://github.com/philsquared/Catch) registered for path 'external/catch'
     Submodule 'external/pegtl' (https://github.com/ColinH/PEGTL) registered for path 'external/pegtl'
     Cloning into 'external/catch'...
     Submodule path 'Realm/ObjectStore/external/catch': checked out 'ce37f48ffad2401e6c23a54cae3f015c01fdc200'
     Cloning into 'external/pegtl'...
     Submodule path 'Realm/ObjectStore/external/pegtl': checked out '3c4128a7e7e1288eb746418ea60c41477358f26a'
 > Running prepare command
   $ /bin/bash -c  set -e sh build.sh cocoapods-setup
   core is not a symlink. Deleting...
   Downloading dependency: sync 1.5.0
  > Copying Realm from
  `/Users/maximumten/Library/Caches/CocoaPods/Pods/Release/Realm/2.5.1-32f86` to
  `Pods/Realm`
  - Running pre install hooks

Generating Pods project
  - Creating Pods project
  - Adding source files to Pods project
  - Adding frameworks to Pods project
  - Adding libraries to Pods project
  - Adding resources to Pods project
  - Linking headers
  - Installing targets
    - Installing target `Realm` iOS 8.0
      - Generating Info.plist file at `Pods/Target Support
      Files/Realm/Info.plist`
      - Copying module map file to `Pods/Target Support
      Files/Realm/Realm.modulemap`
    - Installing target `Pods-MyApp` iOS 8.0
      - Generating Info.plist file at `Pods/Target Support
      Files/Pods-MyApp/Info.plist`
      - Generating module map file at `Pods/Target Support
      Files/Pods-MyApp/Pods-MyApp.modulemap`
      - Generating umbrella header at `Pods/Target Support
      Files/Pods-MyApp/Pods-MyApp-umbrella.h`
  - Running post install hooks
  - Writing Xcode project file to `Pods/Pods.xcodeproj`
    - Generating deterministic UUIDs
  - Writing Lockfile in `Podfile.lock`
  - Writing Manifest in `Pods/Manifest.lock`

Integrating client project

Integrating target `Pods-MyApp` (`MyApp.xcodeproj` project)
  Adding Build Phase '[CP] Embed Pods Frameworks' to project.
  Adding Build Phase '[CP] Copy Pods Resources' to project.
  Adding Build Phase '[CP] Check Pods Manifest.lock' to project.
  - Running post install hooks
    - cocoapods-stats from
    `/Library/Ruby/Gems/2.0.0/gems/cocoapods-stats-1.0.0/lib/cocoapods_plugin.rb`

Sending stats
      - Realm, 2.5.1

Thanks for your help!
Max

Having a hard time finding the other issue. Would you be able to reference which issue # that was?
Thanks

When I googled the error that is the subject of this issue I found https://github.com/realm/realm-object-store/issues/409
I don't know how relevant that is.
In my case I successfully added the Realm Pod and it was working fine, but then I got this error after a git merge of the branch where I had successfully added the Realm Pod.

@MaxMargolis could you please email us an archive of the full Xcode project so we can debug this for you? We can't reproduce this on our end. Thanks!

Thanks for the offer, that's very generous! I'll try and sort it out on my own and if i get totally stuck I'll send it to you.

Thanks a lot, it really works.


SVGs in dark mode are no longer inverted

It does show inverted for me in dark mode, can you confirm if this still exists ?


⚠ Processors has degraded performance

In 15a810d, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 93 ms

Resolved: Processors performance has improved in 637934d.


remove inconsistent axioms

remove the axioms using the IAO_0000039 (has measurement unit label).
@celineaubert the units should already be linked to the qualities. If
the agro qualities are correctly placed in the tree, the units can be
inferred through these links.
if not, add the axioms directly to the units (alternatively open an
issue on the uo tracker)
more fixes to come

removed axioms manually from agro-edit


Can't find older version to install

Hi! I am interested in installing an older version of this plugin (2023.1.*), as is suggested in this comment. What luabud (who I think works on this team?) describes is my situation exactly. However:

my VSCode only lists the version that I am already on
https://marketplace.visualstudio.com/items?itemName=ms-python.debugpy only lists a few recent versions
https://github.com/microsoft/vscode-python-debugger/releases only lists a single version

Is there a way to get my hands on an older version?

Thanks for your issue report, you can follow the steps here in order to install a pre-release version of the extension.


[Bug Report]: HELO not correctly configured when sending emails through SMTP

Describe the Bug
Using a Docker container with Exim for mailing, rejects requests from BookStack as HELO is not correctly announced when opening the SMTP connection.
This was also reported a while ago on issue #2384, but was closed while waiting for more information.
The error the Exim container throws is:
445   rejected HELO from wiki-main-1.mailing [<IP_ADDRESS>]: syntactically invalid argument(s): _

I have this same setup working with other projects which didn't need any special configuration and sent a HELO correctly, allowing them to send email without problems.
Steps to Reproduce

Setup ixdotai/smtp:latest docker image (the one I use).
Setup BookStack on the same Docker network.
Configure the .env file to connect to the SMTP container.
Watch it fail when using the 'Sent test email' button from BookStack. The error is reported on the compose logs from the SMTP container.

Expected Behaviour
BookStack should start the SMTP with a good HELO (maybe the public domain name set on APP_URL could work). An IP is not an option either, since is not a valid HELO.
Screenshots or Additional Context
No response
Exact BookStack Version
v21.11
PHP Version
7.4
Hosting Environment
Ubuntu 20.04, but using docker:
I'll explain my setup:

To explain the part 'wiki-main-1.mailing' of the log:

My SMTP container (where Exim is) is on a Docker network called mailing and every container I want to communicate with it is added to this network. The container is called smtp which later is referenced on the .env from BookStack (since is a DNS resolvable name).
BookStack is on a Compose project called wiki and his container name is main.

The IP <IP_ADDRESS> is the one the BookStack container uses when connected to the 'mailing' network.

This SMTP container is local only so it doesn't have any authentication or SSL termination. This is the configuration set on Bookstack .env:
# Mail sender details
MAIL_FROM_NAME="Wiki"
<EMAIL_ADDRESS>
# SMTP mail options
# These settings can be checked using the "Send a Test Email"
# feature found in the "Settings > Maintenance" area of the system.
MAIL_HOST=smtp
MAIL_PORT=25
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ENCRYPTION=null

Hi @RixzZ,
This can come down to how the webserver is configured within the system.
Are you using the linuxserver.io docker container? If so then the following would be relevant:
https://github.com/linuxserver/docker-bookstack/issues/68

Hi @ssddanbrown,
Yeah, I'm using that container, so that might be the reason it's failing in this case, since the HELO that reaches Exim is _. I guess they need to fix it on the container then. Thank you.
Either way, shouldn't be possible to set it from the application when using SMTP? I think the PHP library used for it might have that ability and it could be a new optional setting on the .env.

Either way, shouldn't be possible to set it from the application when using SMTP? I think the PHP library used for it might have that ability and it could be a new optional setting on the .env.

Yeah, It's likely possible, and we'd probably be able to use the existing APP_URL value, but it'd tie us down to the lower level library implementation while technical be a change in implementation for a much wider set of users, introducing potential risk to wider set of current configurations without spending time understanding or pre-empting how the change may apply. That's upon then maintaining this change for this specific case going forward. For the level of occurance, where this only pops up in container+exim usage, I'm not sure it's worth the change on the app side.
For the linuxserver.io container, pretty sure they expose the nginx config through the /config file you should have mounted via a volume.

Okay @ssddanbrown, thank you for considering it 👍🏼


netopeer2-devel-server not compatible with latest sysrepo-devel

I'm seeing netopeer2 crashed in the callback subscribed by "sr_oper_get_items_subscribe".
As of sysrepo-devel version 1.1.0, the definition for "sr_oper_get_items_cb" has been changed to add request_xpath and request_id to callbacks, but the netopeer2-devel is not updated for this change.
SW version:
netopeer2-server 1.0.1
libsysrepo.so.1.1.2

Hi,
sorry about that, should be updated now.
Regards,
Michal

Thanks Michal, I confirm the latest code is working.


bump to Kotlin 1.9, update kotlin-wrappers for JS target

Hey there!
I'm using buffer with KotlinJS and noticed that the kotlin-wrapper interface for the TextDecoder class has changed recently. If a more recent version of org.jetbrains.kotlin-wrappers:kotlin-js gets pulled into a project (in addition to buffer), this leads to quite the breakage (tm).
Just as an example:
Consider the following dependency block in some arbitrary build.gradle.kts:
val jsMain by getting {
    dependencies {
        val kotlinWrappersVersion = "1.0.0-pre.599"

        implementation("com.ditchoom:buffer:1.3.0")

        implementation(enforcedPlatform("org.jetbrains.kotlin-wrappers:kotlin-wrappers-bom:$kotlinWrappersVersion"))
        implementation("org.jetbrains.kotlin-wrappers:kotlin-js")
        implementation("org.jetbrains.kotlin-wrappers:kotlin-node")

    }
}

Add to that a very simple main method:
fun main() {
    val buffer = PlatformBuffer.allocate(
        69, // nice
        zone = AllocationZone.Direct,
        byteOrder = ByteOrder.BIG_ENDIAN
    )

    buffer.writeByte(0)

    println("Printing the buffer now:")
    println(buffer.readString(69, Charset.UTF8))
}

Then, you'll be greeted with the following error when launching it:
IrLinkageError: Function 'decode' can not be called: No function found for symbol 'web.encoding/TextDecoder.decode|5794919454994727980[0]'
This can be fixed by simply pulling in newer versions of the wrapper when building.
I also took the liberty and bumped the coroutine dependency as well as Kotlin to 1.9 and got rid of some of the deprecation warnings regarding the Android build target.
(I can also split this up or leave it out if desired.)
Thanks for all of your hard work! :)

Hey thanks for the PR, I'll take a look and get back!
(My notifications aren't set correctly so apologies on missing this)

Awesome, thanks!
I just noticed that CI is failing (well, that's embarrassing 😬). Will take a look at this tonight and provide a fix.
Sorry about that.

Hey there!
Apologies for the delay. I've pushed another commit that should hopefully fix the issue. Though, this would now remove support for the old legacy JS compiler.
In a nutshell: It seems that Kotlin 1.9 now throws an error when the old LEGACY JS compiler is used (was deprecated before in 1.8), see: https://kotlinlang.org/docs/js-ir-compiler.html

Looks like the Apple target is causing issues - wasn't able to test that on my machine locally and originally didn't think about this - sorry!
I just enabled GitHub actions on my fork, will work on a fix there and report back with a working build.

Thanks for your contributions @simwagner


[WASM] - Create runtime for WASM Module Compilation - Part 2

We will need to create a WASM specific runtime that compiles all WASM modules in a chunk when loaded Asynchronously and will support parallel chunk loading.

done


add updated ndt-server post

This change is

_posts/blog/2022-02-16-run-a-private-ndt-server.md, line 274 at r1 (raw file):
Previously, stephen-soltesz (Stephen Soltesz) wrote…
This looks incomplete.

It is incomplete as indicated by the ... on the line below it. It's just to show the --restart flag. Maybe this block of code isn't really necessary.

_posts/blog/2022-02-16-run-a-private-ndt-server.md, line 173 at r1 (raw file):
Previously, stephen-soltesz (Stephen Soltesz) wrote…
You might point interested users at https://github.com/m-lab/tcp-info/tree/master/cmd/csvtool to extract some of the data in the JSON.

Can you extract the jsonl files from the .zst archives with csvtool? I hesitate to mention a bunch of tools since people have such wide personal preferences.

_posts/blog/2022-02-16-run-a-private-ndt-server.md, line 173 at r1 (raw file):
Previously, stephen-soltesz (Stephen Soltesz) wrote…
The uncompressed JSON is still effectively unusable. The csvtool provides a dump of the raw data. If a specific example here is too much, you could leave these commands out and/or simply refer the reader to the tcp-info csvtool README for more information.
This is a sample command to decompress a zst file and dump the content.
go get github.com/m-lab/tcp-info/cmd/csvtool
docker run -v $PWD:/data --rm --entrypoint /bin/zstd -it measurementlab/tcp-info:v1.5.3 -cd /data/mlab1-pdx0t_1642181106_000000000000B0C6.00045.jsonl.zst | ~/bin/csvtool

ack. will include.

_posts/blog/2022-02-16-run-a-private-ndt-server.md, line 274 at r1 (raw file):
Previously, stephen-soltesz (Stephen Soltesz) wrote…
ah, okay -- yeah, I would remove this bit and just reference the "Docker command above" in the second bullet above.

done.

@stephen-soltesz Addressed your comments in my latest commits. PTAL one last time?

Thanks for the review @stephen-soltesz


Swap file stays after :wq

Minimal config needed to reproduce
call plug#begin('~/.config/nvim/plugged')
Plug 'mhinz/vim-mix-format'
call plug#end()

au BufReadPost *.ex set filetype=elixir
let g:mix_format_on_save = 1

Neovim version
NVIM v0.3.4
Build type: Release
LuaJIT 2.0.5
Compilation: /usr/bin/cc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -Wconversion -O2 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fdiagnostics-color=auto -Wno-array-bounds -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/build/neovim/src/build/config -I/build/neovim/src/neovim-0.3.4/src -I/usr/include -I/build/neovim/src/build/src/nvim/auto -I/build/neovim/src/build/include
Compiled by builduser

Features: +acl +iconv +jemalloc +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

How to reproduce
Put my minimal config into elixir.vim and run nvim test.ex -u elixir.vim
Insert some elixir code and :wq a bunch (make sure its actually formatting)
Actual swap file error
E325: ATTENTION
Found a swap file by the name "~/.local/share/nvim/swap//%home%peep%.config%nvim%ftplugin%test.ex.swp"
          owned by: peep   dated: Tue Apr 30 21:18:49 2019
         file name: ~peep/.config/nvim/ftplugin/test.ex
          modified: no
         user name: peep   host name: core
        process ID: 8395
While opening file "test.ex"
             dated: Tue Apr 30 21:18:49 2019

(1) Another program may be editing the same file.  If this is the case,
    be careful not to end up with two different instances of the same
    file when making changes.  Quit, or continue with caution.
(2) An edit session for this file crashed.
    If this is the case, use ":recover" or "vim -r test.ex"
    to recover the changes (see ":help recovery").
    If you did this already, delete the swap file "/home/peep/.local/share/nvim/swap//%home%peep%.config%nvim%ftplugin%test.ex.swp"
    to avoid this message.

Swap file "~/.local/share/nvim/swap//%home%peep%.config%nvim%ftplugin%test.ex.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (D)elete it, (Q)uit, (A)bort:

When checking, that process is not running on my system and i continue to get the error. i fixed it in this case with rm ~/.local/share/nvim/swap//%home%peep%.config%nvim%ftplugin%test.ex.swp
EDIT: Was able to reproduce without elixir-editors/vim-elixir

That's a bug in the latest release which is already fixed on master. But I found that the above workaround works for releases up to 3.5! :+1:

I still have the issue with Neovim 0.3.8.

That's a bug in the latest Nvim release which is already fixed on master.

What is the reference of the bug?

But I found that the above workaround works for releases up to 3.5! +1

What is the work-around? I only see deleting the .swp file, but this is not really a work-around as this must be done each time the file is edited after nvim has been exited.


warning: "Legacy editable install is deprecated."

We use local copy of flask-jsonrcp with applied JWT patch (https://github.com/cenobites/flask-jsonrpc/issues/370#issuecomment-2310147193).
Invoking pip install -e /local/path results in this deprecation warning:
DEPRECATION: Legacy editable install of Flask-JSONRPC==3.0.1 from file:///local/path/3rdparty/flask-jsonrpc (setup.py develop) is deprecated. pip 25.0 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457
We tried upgrading setuptools (74.1.2 currently) but id did not helped.
pip is 24.2.
Python 3.9 (as in Debian 11).

The project uses the setuptools as backend to compile Python modules to C extensions using Mypyc[1]. I will see how to migrate to Flit[2] and keep the setuptools only to compile Python modules to C extensions.
Thank you for that heads-up!
[1] - https://github.com/mypyc/mypyc
[2] - https://flit.pypa.io/en/stable/

@Talkless please, check if the fix solve this warning.

Deprecation warning is gone, thanks!