[ADAM-916] New strategy for writing header.

Resolves #916. Not fully done yet; seems to work for BAM but not SAM. The SAM file header doesn't write correctly... Still debugging that. Needs more testing.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1047/Build result: FAILUREGitHub pull request #917 of commit 2c38bbd5ff68bd86ddb00ccb6432b228b1a5b360 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains 8ff91b477e29e28d277a812a87673b43fbbff9a1 # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision 8ff91b477e29e28d277a812a87673b43fbbff9a1 (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 8ff91b477e29e28d277a812a87673b43fbbff9a1First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

This is good to go now. Ping for review and merge.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1048/Build result: FAILURE[...truncated 24 lines...]Triggering ADAM-prb ? 2.3.0,2.11,1.2.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.2.1,centosTriggering ADAM-prb ? 1.0.4,2.11,1.2.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.3.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.10,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.2.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.2.1,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.11,1.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

OK, it seems like on Spark 1.2 we got a merging order that is inconsistent with the ordering we're expecting. I was worried about that... Let me look into this more and see if there is a way we can ensure a consistent merging order.

Rebased and somewhat fixed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1050/Build result: FAILUREGitHub pull request #917 of commit b8ec48ee390d3151f26ffc9095a600b6823ffccf automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 0a51cd7242eb066b795ff1d414b695e3eff0c6b6First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Single unit test failure
Couldn't find any files matching /tmp/bqsr1442238865989322732/bqsr1.bam

Jenkins, retest this please.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1051/Build result: FAILUREGitHub pull request #917 of commit b8ec48ee390d3151f26ffc9095a600b6823ffccf automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 0a51cd7242eb066b795ff1d414b695e3eff0c6b6First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Jenkins, test this please.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1052/Build result: FAILUREGitHub pull request #917 of commit b8ec48ee390d3151f26ffc9095a600b6823ffccf automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 0a51cd7242eb066b795ff1d414b695e3eff0c6b6First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Jenkins, test this please.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1053/Build result: FAILUREGitHub pull request #917 of commit b8ec48ee390d3151f26ffc9095a600b6823ffccf automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision 0a51cd7242eb066b795ff1d414b695e3eff0c6b6 (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 0a51cd7242eb066b795ff1d414b695e3eff0c6b6First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Pushed a fix; tests should pass on this run. From my side, this PR is good to go (assuming tests pass on @AmplabJenkins).

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1054/Build result: FAILURE[...truncated 32 lines...]Triggering ADAM-prb ? 2.3.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.10,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.2.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.2.1,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.11,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Good news: the concat method doesn't exist in org.apache.hadoop.fs.FileSystem in the Hadoop 1.x stream. I've added code that calls concat via reflection. I'm testing this on the cluster now.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1055/Build result: FAILURE[...truncated 32 lines...]Triggering ADAM-prb ? 2.3.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.0,centos completed with result FAILUREADAM-prb ? 1.0.4,2.10,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.2.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.2.1,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.11,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1057/
Test PASSed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1058/Build result: FAILUREGitHub pull request #917 of commit e9cedc15c71d4b983eebc77ddc580a8350890feb automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains 5dade159e8469006b153c4b151caacb23d6f6952 # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision 5dade159e8469006b153c4b151caacb23d6f6952 (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 5dade159e8469006b153c4b151caacb23d6f6952First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

This all works, except for the concat operation, which fails on the cluster because the header doesn't take up an exact, full HDFS block. Anyways, we fall back on the single threaded writing code, which is not so hot from a performance perspective, but works correctly.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1059/Build result: FAILUREGitHub pull request #917 of commit 386f759e195dfbe06882fff7186f4c5211ca49b8 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains 0c8180369152d9447e08c7c89edf1068950c3a00 # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision 0c8180369152d9447e08c7c89edf1068950c3a00 (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 0c8180369152d9447e08c7c89edf1068950c3a00 > git rev-list 5dade159e8469006b153c4b151caacb23d6f6952 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Jenkins, retest this please.

Also, even when we fall back on the single threaded copy code, it takes ~1sec per file to merge the files. E.g., if we have ~2000 partitions, then it takes about 2000 seconds to merge the whole file together. This is unfortunate, but is much better than the ~30hrs latency @beaunorgeot had been seeing when doing something like samtools cat on a single machine.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1060/
Test PASSed.

Thanks for tackling this @fnothaft; I'm missing state on this, can you answer a few high-level questions:

do you know how the missing header issue is currently happening? Last I looked at this (many months ago), we had what we thought was a high-confidence fix, and yet I still saw it happen once (but not reproducibly).
in this fix, you're writing out the header to a path that is specified in the hadoop job config, and having each partition read the header in from that path?
it seems like all the existing {add,clear}Header code is still present, but has been moved around; how does that interact with the new writing/reading-header-file stuff?

OK, this succeeded on the cluster. Save time is slower than I'd like (~40 min to cat NA12878 together) but it worked. I'm running another test now that sorts then saves, then I'll pull the file out of HDFS and run samtools index locally. If that succeeds, then we are good to go. I'll update when that's done; I'm guessing it'll probably finish sometime late tonight.

samtools index worked on the cluster, so we are both writing a concatenated file and keeping the sort order. Yippee!
@ryan-williams

do you know how the missing header issue is currently happening? Last I looked at this (many months ago), we had what we thought was a high-confidence fix, and yet I still saw it happen once (but not reproducibly).

God, I still have no idea what's causing it. I agree—I thought our fix was high confidence, but the log files are pretty unambiguous that we're getting a partition that tries to write to disk but the header is not set.

in this fix, you're writing out the header to a path that is specified in the hadoop job config, and having each partition read the header in from that path?

Exactly. It turns out that this is how Hadoop-BAM's CLI uses the Hadoop-BAM output formats. This isn't obvious though... I got the idea for this just by reading the Hadoop-BAM CLI source code.

it seems like all the existing {add,clear}Header code is still present, but has been moved around; how does that interact with the new writing/reading-header-file stuff?

I only touched the headerless output formats, so they're still vulnerable to the transient header disappearance issue. I'll need to propagate the fix to all output formats later. We'll probably have to do the same thing for VCF as well.

Anyways, this is good to merge now.

@ryan-williams can you review?
I'll take a look and run a few extra tests this morning.  We would like to get this merged as soon as possible.

Unfortunately I don't have much time to get in to the weeds on this one, but my main thought is:
if we don't know why the previous fix didn't fix the issue, even though we all still believe it should have from everything we know about the code, then I don't like complicating matters in this way.
Ultimately, I'll defer to y'all since it sounds like the goal here is more to unblock specific user(s) than to have a fix that we understand, but obviously that doesn't seem like a great state of affairs longer-term, especially if this introduces a slow-down to boot.
Can we deterministically repro the crash? I agree that something is still wrong, @danvk (on #721) and I (undocumented?) have both seen the bug since the last "fix", IIRC.

The perf is only bad when saving BAM as a single file, and "bad" is relative to the FileSystem 'concat' method. This is a correctness fix (block ordering + header attach) relative to the old 'FileUtil.copyMerge' approach. I agree that we need to harmonize the single file save/many file save approaches, but that's for later.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1061/Build result: FAILUREGitHub pull request #917 of commit 7d4b40932275ebf4146fb44d0e03acb3308e0cd9 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/917/merge^{commit} # timeout=10 > git branch -a --contains b12af513d5cb95c54fe156912eef38bc2a37e23c # timeout=10 > git rev-parse remotes/origin/pr/917/merge^{commit} # timeout=10Checking out Revision b12af513d5cb95c54fe156912eef38bc2a37e23c (origin/pr/917/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f b12af513d5cb95c54fe156912eef38bc2a37e23c > git rev-list 0c8180369152d9447e08c7c89edf1068950c3a00 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

This is good to go from my side. @heuermh were you able to complete your testing?
Just to follow up more on @ryan-williams' comments from earlier (sorry, I was on the run so I kept my comments brief), this fix only impacts the single file save code, and does not lead to a performance decrease over the current single file save code. This fix does three things:

Writes the file header to a file in HDFS (or whichever underlying file system you are using; I'll use HDFS as a placeholder here for convenience).
Adds code to the headerless output formats that ensures that they read the header from HDFS. This code will never fail unless writing the header to HDFS failed in the first place, in which case you've got bigger problems and I can't help you ;)
Modifies the file merging code. This introduces a fast path to merging using the concat method, and a second path to merging that manually merges the files together. The second path (manual merge) is necessary anyways, because the prior method using FileUtil.copyMerge did not guarantee the ordering of files that were being merged, which meant that we'd occasionally get files where the header was at the end, etc.

The manual merge code path is much slower than the concat method, but the concat method has harsh, undocumented preconditions (specifically, all files being merged—except for the last one—must take up a full block in HDFS) and seems to throw a NotImplementedError on file systems that aren't HDFS, which is dandy. The manual merge code path is not slower than the FileUtil.copyMerge approach we were using previously.
As for the million dollar question: why does the header detach issue keep on happening, despite our best efforts to fix it? Frankly, I'm not entirely sure why it's failing, but the root cause is something along the lines of "when we set the header in the singleton object that accompanies the output format, it doesn't get consistently set across all of the JVMs where an instance of the actual output format will be created and executed." The new approach uses the Hadoop job configuration to pass the path to the header file. This conf is passed to the OutputFormat by Spark itself, so we can guarantee that the configuration seen by the OutputFormat will be consistent across the cluster. Since the header is written to the underlying file system (which again, if your underlying file system is not accessible across the whole cluster, then you're running some horrible frankencluster and I can't help you), we can guarantee that reading the header file from the path passed in the config will succeed.
I think a TL;DR for why this failed is that the obvious way of attaching the header (I call it obvious because the method is named setHeader) to Hadoop-BAM's OutputFormats is not guaranteed to work consistently in a distributed setting. I think that the Hadoop-BAM team knows this, because the Hadoop-BAM CLI wraps their own output formats to do this exact "write header to HDFS, set path in config, read header from config path when creating record reader" pattern.

Jenkins, retest this please.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1062/
Test PASSed.

Tested ok for me, merged from the ski lift.
Thank you, @fnothaft!


Cleanup for group

Description:
Bad syntax (IMO)

is None is a very fast pointer comparison. not X means we're going to call __bool__ on that object, resulting in a lot more function calls.
I don't consider this a clean up…

I do agree to this, did not think about that.. sorry


Add conditional rpc example

Appreciate the PR! Let's keep the docs simple for now. Good news is this PR will turn up in search results for others so is still documenting.


Fixed Issue 16387 - getSymbolsByUDA works with structs but fails with classes

https://issues.dlang.org/show_bug.cgi?id=16387
(getSymbolsByUDA works with structs but fails with classes)

added a filter to skip the inaccessable 'this' member

:white_check_mark:


Quic-Go team, are you ok?

Hello,
I m big fan of quic-go.
I check project quic-go 2-3 times per day.
I saw quic-go have no new commit since 7 days ago.
Marten-seemann still work in Action and Pull Request review for lucas-clemente, but lucas-clemente no reply.
I have a bit worries.
Is Quic-Go team/Mr.lucas-clemente ok?
I hope all you well.

@okgame, I can assure you that both me and @lucas-clemente are fine. As you can see from the large number of open PRs, there will be a bunch of progress on this repository very soon.

@okgame, I can assure you that both me and @lucas-clemente are fine. As you can see from the large number of open PRs, there will be a bunch of progress on this repository very soon.


Enforce challenge submission order

So far we're allowing ppl to submit any challenge without taking into account the order.
@carletex I think it's a bug, but feel free to update the tag. Not sure if Austin mentioned something about allowing ppl to submit challenges in any order.

We haven't defined a logic for it yet. I'll ask Austin!


pnum_tensor property missing

In hvd_simd_mesh_impl.py  the laid out tensor function that requires a property .pnum_tensor. In the standard simd_mesh_impl.py this is defined on line 87, but there is not an equivalent in hvd_simd_mesh_impl.py. This raises problems when random is called.
Below is an example of this happening:
    lowering = mtf.Lowering(graph, {mesh: mesh_impl})
  File "<...>/mesh/mesh_tensorflow/ops.py", line 728, in __init__
    op.lower(self)
  File "<...>/mesh/mesh_tensorflow/ops.py", line 5799, in lower
    mesh_impl.random(output_shape, self._tf_fn, self._kwargs)))
  File "<...>/mesh/mesh_tensorflow/hvd_simd_mesh_impl.py", line 607, in random
    tf.equal(self.laid_out_pcoord(axis).one_slice, 0), x.dtype)
  File "<...>/mesh/mesh_tensorflow/ops.py", line 1209, in laid_out_pcoord
    return self.slicewise(my_fn, self.laid_out_pnum())
  File "<...>/mesh/mesh_tensorflow/hvd_simd_mesh_impl.py", line 268, in laid_out_pnum
    return self.LaidOutTensor([self.pnum_tensor])
AttributeError: 'HvdSimdMeshImpl' object has no attribute 'pnum_tensor'

this should be solved


python39Packages.pysigma: 0.5.0 -> 0.5.1

Description of changes
https://github.com/SigmaHQ/pySigma/releases/tag/v0.5.1

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 170589 run on x86_64-linux 1

13 packages built:

python310Packages.pysigma
python310Packages.pysigma-backend-insightidr
python310Packages.pysigma-backend-splunk
python310Packages.pysigma-pipeline-crowdstrike
python310Packages.pysigma-pipeline-sysmon
python310Packages.pysigma-pipeline-windows
python39Packages.pysigma
python39Packages.pysigma-backend-insightidr
python39Packages.pysigma-backend-splunk
python39Packages.pysigma-pipeline-crowdstrike
python39Packages.pysigma-pipeline-sysmon
python39Packages.pysigma-pipeline-windows
sigma-cli

Result of nixpkgs-review pr 170589 run on x86_64-linux 1

13 packages built:

python310Packages.pysigma
python310Packages.pysigma-backend-insightidr
python310Packages.pysigma-backend-splunk
python310Packages.pysigma-pipeline-crowdstrike
python310Packages.pysigma-pipeline-sysmon
python310Packages.pysigma-pipeline-windows
python39Packages.pysigma
python39Packages.pysigma-backend-insightidr
python39Packages.pysigma-backend-splunk
python39Packages.pysigma-pipeline-crowdstrike
python39Packages.pysigma-pipeline-sysmon
python39Packages.pysigma-pipeline-windows
sigma-cli


[Question]: What would be needed to add hexapod mark 3 support ?

Question
What would be needed to add hexapod mark 3 support ?
I know I can just place the required urdf and stl files but the mark 3 uses DYNAMIXEL AX-12A servos.
I would like to know what changes would be needed to add support for the mark 3.
Robot Model
mark3
Operating System
Ubuntu 20.04
ROS Version
ROS1 Noetic
Anything Else
No response

@LSinterbotix @swiz23 @mtrossen small follow up, I was wondering what changes would be needed to get it working on a mark 3 with AX-12A servos

The base ROS interface to the motors is through the interbotix_xs_sdk. The core wraps the dynamixel_workbench_toolbox. According to ROBOTIS, the toolbox is compatible with the AX-12 servos.
The SDK is designed for the X-Series DYNAMXIELs, so much of its functionality is built up with the assumption that we'd be using those more advanced interfaces. You would have to strip away a lot of that functionality like managing operating modes and change how things like joint limits are managed. The AX-12A motors seem to only operate in a position-based mode with timing controlled by the Moving Speed register. On top of that, you will need to modify the Python-ROS API to reflect the changes made to the SDK.
There are probably other things you would need to change to get full compatibility, but this should be a good starting point if you're interested in undertaking this effort. I'll be closing this issue as it's unrelated to the officially supported ROS-based crawlers, but feel free to continue to ask questions in this thread.


coredump in Ubuntu20.04/x86-64

how to reproduce?
step 1,
rustup component add rust-src llvm-tools-preview
step 2,
run the command in https://github.com/rcore-os/rCore#how-to-run:
$ git clone https://github.com/rcore-os/rCore.git --recursive
It workks well
$ cd rCore/user
$ make sfsimg prebuilt=1 arch=x86_64
It workks well too
$ cd ../kernel
$  make run ARCH=x86_64 LOG=trace
// ...
warning: unused import: lazy_static::*
--> src/memory.rs:24:5
|
24 | use lazy_static::*;
|     ^^^^^^^^^^^^^^
|
= note: #[warn(unused_imports)] on by default
warning: unnecessary parentheses around if condition
--> src/sync/mutex.rs:150:12
|
150 |         if (initialization == 2) {
|            ^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
|
= note: #[warn(unused_parens)] on by default
warning: unnecessary parentheses around if condition
--> src/sync/mutex.rs:153:12
|
153 |           if (initialization == 1
|  ^
154 | |             || self
155 | |                 .support_initialization
156 | |                 .compare_and_swap(0, 1, Ordering::Acquire)
157 | |                 != 0)
| |_________^
|
help: remove these parentheses
|
153 |         if initialization == 1
154 |             || self
155 |                 .support_initialization
156 |                 .compare_and_swap(0, 1, Ordering::Acquire)
157 |                 != 0
|
warning: use of deprecated item 'sync::condvar::Condvar::_wait': this may leads to lost wakeup problem. please use wait instead.
--> src/sync/mutex.rs:318:14
|
318 |         self._wait();
|              ^^^^^
|
= note: #[warn(deprecated)] on by default
// ...
error: "/root/.rustup/toolchains/nightly-2020-04-06-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/Cargo.lock" does not exist, unable to build with the standard library, try:
rustup component add rust-src
make: *** [Makefile:301: kernel] Error 101
// ...
make[1]: 离开目录“/root/桌面/rCore/rboot”
qemu-system-x86_64: -device ide-drive,drive=sfsimg,bus=ahci0.0: warning: 'ide-drive' is deprecated, please use 'ide-hd' or 'ide-cd' instead
I run the command in step 1 again,
then I change ide-drive to ide-hd:
https://github.com/rcore-os/rCore/blob/a838a1fb9aad45f68b3599a76d8a86ab0ecfb485/kernel/Makefile#L94
and remove the start @ in justrun:
https://github.com/rcore-os/rCore/blob/a838a1fb9aad45f68b3599a76d8a86ab0ecfb485/kernel/Makefile#L248
and then run make run ARCH=x86_64 LOG=trace again, it shows:
qemu-system-x86_64 -smp cores=4 -bios ../rboot/OVMF.fd -drive format=raw,file=fat:rw:target/x86_64/release/esp -serial mon:stdio -m 4G -device isa-debug-exit -drive format=qcow2,file=../user/build/x86_64.qcow2,media=disk,cache=writeback,id=sfsimg,if=none -device ahci,id=ahci0 -device ide-hd,drive=sfsimg,bus=ahci0.0 -nographic
as above, it just block here, and show nothing.
then I type ctrl + a, and then type c, then it shows:
QEMU 4.2.0 monitor - type 'help' for more information
(qemu) quit  # type quit and enter
make: *** [Makefile:248: justrun] 段错误 (core dumped)
but Qemu works well with my own images, and all the related toolchains are latest.
Could you give some help ?

Update QEMU to 5.0?

the first step you said is important, it works well with it


How to add dnscrypt server to server list?

Hello,
This my Details for the UK DNSCRYPT server.
Stamp for dnscrypt-proxy 2.x:
Server name :- dnscrypt.co.uk
Provider name :- 2.dnscrypt-cert.dnscrypt.co.uk
FQDN :- dnscrypt.co.uk >> coming soon 💯
IPv4 address :- <IP_ADDRESS>
IPV6= 2a01:a500:375:0001:0000:0000:39a7:f5e6
-------------------------------- ------------------------------------------------------- sdns://AQcAAAAAAAAAEjE3OC4xNTkuMTAuMTE1OjQ0MyDRBy6TkwUZoKbHspdDR6U5kBcz2kDKXrSkvxcZMX4EtB4yLmRuc2NyeXB0LWNlcnQuZG5zY3J5cHQuY28udWs
DNSCrypt server resolver hosted in London.
DNSSEC validation for better security, Caching, Non-logging, IPV6 Support. Fast and secure.
ps> this server is Unlimited bandwidth 🥇
if you need and more info please type here .
//Ammy

Hi Ammy,
Looks like the dnscrypt-wrapper proxy works, but the resolver itself doesn't return anything.
Can you take a look?

Hi.
What resolver?

May it be a firewall issue?
If you are using this image, the resolver is unbound.
Are you able to use it with this configuration?

I can.t use it

:(

That's why i need your help

how to modify unbound, i can't find it? please little bit help with this

Can you try the instructions here? https://github.com/jedisct1/dnscrypt-proxy/wiki/Manually-updating-your-DNSCrypt-server

I did trying the instructions , what now?

Congratulations! The container has been properly initialized.
Take a look up above at the way dnscrypt-proxy has to be configured in order
to connect to your resolver. Then, start the container with the default command.
Starting DNSCrypt service for provider:
2.dnscrypt-cert.dnscrypt.co.uk
Starting pre-service scripts in /etc/runit_init.d
Generate crypt key pair... ok.
Secret key stored in /opt/dnscrypt-wrapper/etc/keys/short-term/1561824553.key
[53] 29 Jun 16:09:13.198 [notice] [main.c:673] Generating pre-signed certificate.
[53] 29 Jun 16:09:13.198 [notice] [main.c:680] TXT record for signed-certificate:

Record for nsd:
[53] 29 Jun 16:09:13.198 [notice] [main.c:695] Certificate stored in /opt/dnscrypt-wrapper/etc/keys/short-term/1561824553.c      ert.
2.dnscrypt-cert 86400   IN      TXT     "DNSC\000\002\000\000\189|\019M>\185\166\189U\142/b\008\154\147\231\218cg\220\246\1      67N\180rL\2420\031\012\150\174\150\156\148\252do\024\134\217YG\204pS\145\1920B\195\029\127x\145\092m\029\001~\203T\021\002'      ;\203\155\156\228\146\030\153\195\135{1]2\137Q\031\187\134\020\224\255R\235\173A\014e+\0019(;\203\155\156\228\146\030]\023\      141)]\023\141)]\024\222\169"

Record for tinydns:
'2.dnscrypt-cert:DNSC\000\002\000\000\275|\023M\076\271\246\275U\216/b\010\232\223\347\332cg\334\366\247N\264rL\3620\037\01      4\226\256\226\234\224\374do\030\206\331YG\314pS\221\3000B\303\035\177x\221\134m\035\001~\313T\025\002';\313\233\234\344\222      \036\231\303\207{1]2\211Q\037\273\206\024\340\377R\353\255A\016e+\0019(;\313\233\234\344\222\036]\027\215)]\027\215)]\030\3      36\251:86400

setup in directory /opt/unbound/etc/unbound
generating unbound_server.key
Generating RSA private key, 3072 bit long modulus (2 primes)
.................................++++
..........................++++
e is 65537 (0x010001)
generating unbound_control.key
Generating RSA private key, 3072 bit long modulus (2 primes)
...........................................................................................................................      ........................................................++++
...........................................................................................++++
e is 65537 (0x010001)
create unbound_server.pem (self signed certificate)
create unbound_control.pem (signed client certificate)
Signature ok
subject=CN = unbound-control
Getting CA Private Key
Setup success. Certificates created. Enable in unbound.conf file to use
./run: line 102: /proc/sys/vm/drop_caches: Read-only file system
Error: error sending query: Could not send or receive, because of network error
Error: error sending query: Could not send or receive, because of network error
timeout: run: unbound: (pid 31) 315s
./run: line 102: /proc/sys/vm/drop_caches: Read-only file system
Error: error sending query: Could not send or receive, because of network error
Error: error sending query: Could not send or receive, because of network error
kill: run: unbound: (pid 84) 15s
./run: line 102: /proc/sys/vm/drop_caches: Read-only file system
ok: run: dnscrypt-wrapper: (pid 32) 362s

still error if you check in the end and i can't find unbound.

I bought a Vultr instance to try to see what was wrong. Following the instructions reproduced the issue you were seeing. Everything starts, but Unbound can't send anything.
Looks like this is a firewall issue. I never use CentOS (as recommended by the guide), so I don't know if this is a new issue in CentOS, or something specific to Vultr, but anyway, I did the following and everything started to work.
Edit /etc/sysconfig/iptables and replace its content with the one from that file.
Then, type: systemctl reload iptables
And everything should start working :)
I updated the guide with these instructions. Hopefully this will work for you, too.

Thanks for the reply.
I delete my instance and will buy another one with scale way and see how it works
Thanks

installed,
how to add server name if you  still didn't updated mine in server list. or should i use only my stamps to check if its work or not

Stamp for dnscrypt-proxy 2.x:
sdns://AQcAAAAAAAAAEDE3OC4xNTkuMi43MDo0NDMgGBJx_uhd4uH8S7ewYLY9omhAKCHwwI9pXHWm7l_B8OEfMi5kbnNjcnlwdC1jZXJ0LmRuc2NyeXB0LmNvLW5zMA

error on my another server, just tried to install and test dnscrypt
./dnscrypt-proxy -service install
./dnscrypt-proxy: line 30: server_names: command not found
./dnscrypt-proxy: line 36: listen_addresses: command not found
./dnscrypt-proxy: line 41: max_clients: command not found
./dnscrypt-proxy: line 55: ipv4_servers: command not found
./dnscrypt-proxy: line 58: ipv6_servers: command not found
./dnscrypt-proxy: line 61: dnscrypt_servers: command not found
./dnscrypt-proxy: line 64: doh_servers: command not found
./dnscrypt-proxy: line 70: require_dnssec: command not found
./dnscrypt-proxy: line 73: require_nolog: command not found
./dnscrypt-proxy: line 76: require_nofilter: command not found
./dnscrypt-proxy: line 79: disabled_server_names: command not found
./dnscrypt-proxy: line 88: force_tcp: command not found
timeout: invalid time interval ‘=’
Try 'timeout --help' for more information.
./dnscrypt-proxy: line 111: keepalive: command not found
./dnscrypt-proxy: line 118: refused_code_in_responses: command not found
./dnscrypt-proxy: line 148: cert_refresh_delay: command not found
./dnscrypt-proxy: line 193: fallback_resolver: command not found
./dnscrypt-proxy: line 199: ignore_system_dns: command not found
./dnscrypt-proxy: line 209: netprobe_timeout: command not found
./dnscrypt-proxy: line 220: netprobe_address: command not found
./dnscrypt-proxy: line 233: log_files_max_size: command not found
./dnscrypt-proxy: line 236: log_files_max_age: command not found
./dnscrypt-proxy: line 239: log_files_max_backups: command not found
./dnscrypt-proxy: line 253: block_ipv6: command not found
./dnscrypt-proxy: line 291: cache: command not found
./dnscrypt-proxy: line 296: cache_size: command not found
./dnscrypt-proxy: line 301: cache_min_ttl: command not found
./dnscrypt-proxy: line 306: cache_max_ttl: command not found
./dnscrypt-proxy: line 311: cache_neg_min_ttl: command not found
./dnscrypt-proxy: line 316: cache_neg_max_ttl: command not found
./dnscrypt-proxy: line 326: [query_log]: command not found
./dnscrypt-proxy: line 335: format: command not found
./dnscrypt-proxy: line 352: [nx_log]: command not found
./dnscrypt-proxy: line 361: format: command not found
./dnscrypt-proxy: line 382: [blacklist]: command not found
./dnscrypt-proxy: line 410: [ip_blacklist]: command not found
./dnscrypt-proxy: line 438: [whitelist]: command not found
./dnscrypt-proxy: line 472: [schedules]: command not found
./dnscrypt-proxy: line 511: [sources]: command not found
./dnscrypt-proxy: line 515: [sources.public-resolvers]: command not found
./dnscrypt-proxy: line 516: urls: command not found
./dnscrypt-proxy: line 517: cache_file: command not found
./dnscrypt-proxy: line 518: minisign_key: command not found
./dnscrypt-proxy: line 519: refresh_delay: command not found
./dnscrypt-proxy: line 520: prefix: command not found
./dnscrypt-proxy: line 544: [static]: command not found
./dnscrypt-proxy: line 546: [static.dnscrypt.co-ns0]: command not found
./dnscrypt-proxy: line 547: stamp: command not found

Meanwhile, I was trying to understand how to install it on Vultr.
If you still have the Vultr instance, can you try to run the following script? https://gist.githubusercontent.com/jedisct1/b8393a058ba3cfc1335379c4ac41969f/raw/9600ddbc354992b3e988d0f14f12e9983ab102c4/centos5-docker-dnscrypt-server.sh

You can also use it as an install script. I updated the instructions here: https://github.com/jedisct1/dnscrypt-proxy/wiki/DNSCrypt-server-with-vultr.com
And verified that they work on a new as well as on an existing Vultr instance.

I have VPS with quickhostuk, I’ll try to install it using your script you provided and I’ll install DNSCRYPT V2 on another VPS and use the stamp that generated on the VPS CentOS, is that all? Still how can I use my own server name in server name toml file of DNSCrypt V2 using my own DNSCrypt in that shows on dns leak test.
Thank you for the effort you make to make the dns more secure, don’t know how to repay you
Thanks

The script starts with:
SERVER="$(hostname)"

You can replace this with:
SERVER="your host name"

You mean replace it in toml file
My server name for example “ DNSCrypt.co-uk , that I will add it in server name instead of , Cloudflare, google etc
And I’ll add it to static “ deleting google and replace DNSCrypt.co-uk and I’ll put the stamp that I generated in VPS CentOS
If I got it right?

Test is as a static server, and if it works, it can be added with any name tou want (as long as it is unique) in the public-resolvers.md file.

Ok will test it as static without adding anything, and see if it’s work.
After that can I do it by myself I mean add the server name into public-resolver.md file
Of course I’ll check to be unique name.
Or should I contact you to add the server name ?

Last question.
Let’s I setup CentOS 7 and install docker and then start docker and after that I run you bash scripts that you provided me earlier ( Vultr scripts ) and then I stamp and cert , public key “ everything have generated “ on the VPS . How to rest it and see The DNSCrypt-server is working and nothing wrong with resolver before I use it on another VPs
Thanks again

Hello,
How to test the server if its working or not. I used your latest script you provided yesterday . and wanna test it but how?
Generate provider key pair... ok.
Stamp for dnscrypt-proxy 2.x:
sdns://AQcAAAAAAAAAEDE3OC4xNTkuMi43MDo0NDMgvMqXpsXJwIu4SwfwJznbNGyHGHuGxHFPHap                               gvWxna9AfMi5kbnNjcnlwdC1jZXJ0LmRuc2NyeXB0LnVrLW5zMA
Parameters for dnscrypt-proxy 1.x:
dnscrypt-proxy --provider-key=BCCA:97A6:C5C9:C08B:B84B:07F0:2739:DB34:6C87:187                               B:86C4:714F:1DAA:60BD:6C67:6BD0
--resolver-address=<IP_ADDRESS>:443
--provider-name=2.dnscrypt-cert.dnscrypt.uk-ns0
Keys are stored in public.key & secret.key.

Hello,
I tired this one on scaleway .
Stamp for dnscrypt-proxy 2.x:
sdns://AQcAAAAAAAAAEDEwLjE4LjE1MC4yNTo0NDMgMQ5gio0GwQV2eHdV59Sq7ztdQxaqcqn3r3XyQkfmfjUfMi5kbnNjcnlwdC1jZXJ0LnNydi5kbnNjcnlwdC5ubA
Parameters for dnscrypt-proxy 1.x:
dnscrypt-proxy --provider-key=310E:608A:8D06:C105:7678:7755:E7D4:AAEF:3B5D:4316:AA72:A9F7:AF75:F242:47E6:7E35
--resolver-address=<IP_ADDRESS>:443
--provider-name=2.dnscrypt-cert.srv.dnscrypt.nl
Keys are stored in public.key & secret.key.

I think this working , but i don't how Lol
Can you test it, and im using private IP , as your script picked it up


The 1.3.0 release broke the docs page

Looks like there was a step missed in the build process. The minified ui-bootstrap-tpls file was not generated.

why was this closed? Issue is still present I think?

@storkme, did you try forcing a refresh of the page and/or restarting your browser?

My apologies, think it got cached by something in my network! All good now.

@storkme no worries - we've seen some issues for those behind a proxy.  thanks.

this is now fixed - even with an adblocker installed.

Sorry for closing without an explanation. Minutes after I opened the issue it was resolved, so I just chalked it up to someone still in the process of building, etc. I can verify that my problem was not because of caching.


Prepare for testing multiple ISAs in CI

Update matrix structure and add sed commands to change the extensions being tested. Going forward all that needs to be done to test different sets of extensions is add an entry to the matrix.

The following dummy WORKFLOW that I have created for a scenario in which one of the RV32 tests causes the RISCOF run to fail but the job will still continue and upload the artifacts. For RV64, I have made the job to succeed, hence, in both cases, uploading the artifacts successfully.
https://github.com/MuhammadHammad001/riscv-arch-test/actions/runs/11416510012

Furthermore, I have made the CI to mark the job still as a failure in case the riscof run fails, @jordancarlin  please let me know if you want this to be changed!

I doubt this approach will work. We are trying to improve the CI to run multiple extensions and different architecture configurations, like e.g, we want to run misaligned_load tests (in privilege folder) twice, once with misaligned support enabled on both Spike and Sail, and once with misaligned support disabled for both, similarly there will be alot of other tests in future which will require to be run twice based on different configuration options. Such options cannot be passed via CI so we require multiple files in which we will change the configuration flags and manage those files. CI will take those files and run it.
Your work, although is brilliant, but will greatly complicate the CI, my recommendation would be to keep CI simple, and execute a bunch of files in the CI, and maintain those files for longer run to enable/disable the extension and/or to add more sail/spike configuration options running ACTs


Readme outdated for VS 2017

Seems the readme has wrong information about creating a build for VS 2017. I'm talking about this part:
https://gyazo.com/fcce3c02e8c067e0479c929e058d2e85
Could you change requirements from Clang 6 to Clang 7 and change line
cmake -T"LLVM-vs2014" -G "Visual Studio 14 2015 Win64" ../..
to
cmake -T"LLVM" -G "Visual Studio 15 2017 Win64" ../..
Or just add separate 2017 instructions apart from the 2015 ones
It took me quite a while to figure out how to create the build for VS 2017 :)

It should be an addition, not a replacement. We still rely on VS2015.

Are you going to upgrade to VS 2017?
Because when I try to download VS 2015 from the official website  I get the "Sorry, we couldn't find any downloads for you." message, so I think I would not be able to downgrade anymore. That and more and more people are using VS 2017 these days, so I think it would be useful to add a separate tutorial for VS 2017 or to upgrade to 2017.

According to https://github.com/google/filament/issues/639#issuecomment-447593083 the README should say this for VS 2017:
cmake -T"LLVM" -G "Visual Studio 15 2017 Win64" -DCMAKE_CXX_COMPILER:PATH="C:\Program Files\LLVM\bin\clang-cl.exe" -DCMAKE_C_COMPILER:PATH="C:\Program Files\LLVM\bin\clang-cl.exe" -DCMAKE_LINKER:PATH="C:\Program Files\LLVM\bin\lld-link.exe" ../..


It seems that the file "rival_custom_test.sh" is not available in the repo

I'm sorry for not getting back to you sooner.
There is a typo in the README file. Please try the script rival_dreambooth_test.sh instead.


Terms, vacancies, backup signers

Not yet ready for editor attention

Reminder of process for updating RPIP-10, per RPIP-10

An author SHALL make edits as desired and create a pull request; the status in the PR SHALL be "Draft"
An RPIP editor SHALL review it. If there are no structural issues, the status shall be changed to "Review"
The vote procedure SHALL be followed as if this were an entirely new RPIP.
If criteria for a vote are not met, the status SHALL be changed to "No Vote Taken" and the PR SHALL be closed.
If there was a vote, then after the vote is concluded:
If the vote passed, the status SHALL be changed to "Living" and the PR SHALL be merged
If the vote failed, the status SHALL be changed to "Vote Failed" and the PR SHALL be closed

I'm now comfortable with step 1, and am ready for an RPIP editor to do step 2 (or comment they're good with status changing to review and I'm happy to execute).


Run npm install by default

I think it would be useful if demeteorizer could run npm install automatically, either by default or by supplying some argument.
Currently, after running demeteorizer, I have to run:

cd demeteorizer/programs/server
npm install
cd ../..

This is easily achievable by wrapping demeteorizer on a bash script, but it might be useful to have it built in.

This is outside of the scope of demeteorizer.


Overview team in project — crew integration

What:

team title
people count
link to team page

How:

field with team id in project settings

Crew API for getting teams (with search) - https://github.com/taskany-inc/crew/issues/432
Create a combobox on project settings page to link the project to a crew team
Inherit linked team for child projects by default, but support ovveride
Show quick summary about the team


Update 1password-beta to 6.6.3.BETA-2

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask’s name and version.

Oops, looks like this is a duplicate of @3455.


proxy protocol: Add http support.

Signed-off-by: Weston Carlson<EMAIL_ADDRESS>
Commit Message: Add http proxy proto support.
Additional Description: The main addition here is setting the proxy protocol filter state information so that the upstream proxy proto transport socket can use that info to properly generate the header.
Risk Level: medium?
Testing: functional, integration
Docs Changes: Added details to IP Transparency page for using proxy proto with HTTP
Release Notes: Added note for http proxy proto support
Fixes https://github.com/envoyproxy/envoy/issues/15463

cc @ggreenway who is looking at related proxy proto issues.
As I mentioned in the linked issue, I don't think this change by itself makes sense. Because of how Envoy decouples the upstream connection pool from the downstream connection and does keep alive, this change will end up sending requests for different clients to an upstream connection that has the wrong proxy proto already sent. The only way this makes sense is if we have a dedicated connection pool / connection for each unique downstream client. Iirc this is actually possible with various config options, but imo at minimum we need to document this, and even then it's so prone to breaking I would rather enforce in code somehow.

I agree there are a lot of caveat's, but it's neccessary if one doesn't control the upstream, and the upstream expects PROXY protocol.
Definitely need good docs at a minimum. Ideally, there would be a mechanism to close the upstream connection(s) when the downstream connection closes. I'm not exactly sure how this would be implemented. An alternative is documenting (or enforcing) and low idle timeout on the upstream connections. This would be a case where global config validation (not per-filter) is useful.

Shouldn't the connection pool be unique for clients due to the proxy proto header info being in the hash? i.e. https://github.com/envoyproxy/envoy/pull/13768/files#diff-54d8fe86c68f5ce79e1ebcf1994b13096632b8897b26591910b86a2b05e50064R47.
Agree on it being ideal if upstream connection was closed when downstream was closed. I could potentially look into that more but would also be open to the low timeout / documentation approach. Just wanted to confirm my understanding, we don't have a global validator currently, right?

Ok sorry I forgot that the proxy proto setting already effects the hash. Part of me wonders of this change should be config guarded, which could then be used to pivot on extra checks? I have a few other concerns but let me follow up offline with @ggreenway first offline.

Sorry about dismissing your review @alyssawilk, I found one more grammar issue in the docs.

I recently discovered #16682, which will also apply to http with upstream PROXY protocol. I think we should either document that this new code has this defect until that issue is fixed, or wait for the fix for that issue.

My vote would be to document the defect and merge this for now. Mainly because the defect already exists (and for more than just proxy protocol, but it's probably the most affected) and this code mainly just fixes the proxy protocol header sent using http connection manager currently. As noted by other issues, people are trying to use this socket with http connection manager already and noticing the addresses are wrong. I could see the argument against this though being having it in the release notes might means wider adoption and more people hitting this issue.
We (Digitalocean) will likely want the memory issue fixed in the near future as well though. I could potentially look into resurrecting the idle timeout PR that would have fixed the issue, but I don't want to step on anybody's toes.

I am working on a fix for the leak right now. I'll should have a PR within a week.
I'm fine with merging this with extra docs noting the defect. Can you find a good spot where the note will be seen? Maybe also include it in the release note, and assuming this is fixed before the next release (very likely), we can edit that part out of the release note. Seem reasonable?

SGTM!

Ready for another look @ggreenway. I've added defect notes to the proxy protocol section that mentions the socket. Since it mainly affects pp, I thought this would be okay. I'm not sure of a more generic place to document this issue. Also added a link to the issue in the release notes.

Still LGTM!  Matt you OK merging with the defect given docs?

Thanks everyone for taking the time to review!


How to have a nodeJs running on top of the jrottenberg image?

I would need to have my nodejs app runing on top of the jrottenberg image.
Here my docker file:
FROM node:18-alpine as builder

WORKDIR /usr/src/app

COPY package.json yarn.lock ./

RUN yarn install --frozen-lockfile

COPY . .

RUN yarn build

FROM jrottenberg/ffmpeg:4.4-nvidia AS FFmpeg

RUN chown root:root -R /usr/local

FROM node:18-alpine

COPY --from=FFmpeg /usr/local /usr/local

WORKDIR /usr/src/app

ENV PATH="${PATH}:/usr/local/bin"

COPY package.json yarn.lock ./

RUN yarn install --production --frozen-lockfile

COPY --from=builder /usr/src/app/dist ./dist

EXPOSE 4343

CMD [ "node", "dist/index.js" ]

The image is correctly built but I got this error on my nodejs app
Error: spawn /usr/local/bin/ffprobe ENOENT
The ffprobe binary is in  /usr/local/bin/. The PATH includes "/usr/local/bin/".
Even when doing /usr/local/bin/ffprobe [arguments] directly from my VM it doesn't work.
I don't know what else to check..

got it as :
FROM jrottenberg/ffmpeg:6-alpine AS ffmpeg
FROM node:18-alpine
COPY --from=ffmpeg / /

# + ... npm ci etc


Fix lingering_data_frame when data length is 0

Right this must correspond to this that I've seen in Gun:
<EMAIL_ADDRESS>I don't think the lingering data tuple should be returned at all if the length is 0 to be honest.

Done by not returning the lingering_data tuple instead. Thanks!


Dynamically provisioned volumes not properly mounted

After a successfully provisioning the volume, PVC is not mounted properly inside the pod,
following error is visible:
Normal   SuccessfulAttachVolume  2m17s      attachdetach-controller                   AttachVolume.Attach succeeded for volume "pvc-c01d67e2-7018-49a2-ac2d-b42321666f71"
Normal   SuccessfulAttachVolume  2m17s      attachdetach-controller                   AttachVolume.Attach succeeded for volume "pvc-78e0a1ff-edcc-4386-a133-4de50c2ec942"
Warning  FailedMount             15s        kubelet, node1  Unable to attach or mount volumes: unmounted volumes=[data data1], unattached volumes=[domain-service-account-secret control-agent-secrett var-run ]: timed out waiting for the condition
Warning  FailedMount             9s         kubelet, node1  MountVolume.MountDevice failed for volume "pvc-78e0a1ff-edcc-4386-a133-4de50c2ec942" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning  FailedMount             8s         kubelet, node1  MountVolume.MountDevice failed for volume "pvc-c01d67e2-7018-49a2-ac2d-b42321666f71" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Volumes provisioned using ontap-san driver (iscsi)
Environment

Trident version: 20.01.1
Trident installation flags used:   -d -n trident-storage --csi --use-custom-yaml]
Container runtime: 19.03.8
Kubernetes version: v1.17.4
Kubernetes enabled feature gates:
OS: RHEL 7.7
NetApp backend types: ontap-san

To Reproduce
Appears randomly during creating a deployment
Expected behavior
Volume mounts every time
Additional context

May 27 13:09:14 node1 kubelet: I0527 13:09:14.990977    2985 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-c01d67e2-7018-49a2-ac2d-b42321666f71" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-c01d67e2-7018-49a2-ac2d-b42321666f71") pod "control-p4tdq" (UID: "5db708ba-e3e5-4aab-85ea-919c61a9117e")
May 27 13:09:14 node1 kubelet: E0527 13:09:14.991086    2985 nestedpendingoperations.go:270] Operation for ""kubernetes.io/csi/csi.trident.netapp.io^pvc-c01d67e2-7018-49a2-ac2d-b42321666f71"" failed. No retries permitted until 2020-05-27 13:09:15.491057427 +0200 CEST m=+732787.128930660 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume "pvc-c01d67e2-7018-49a2-ac2d-b42321666f71" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-c01d67e2-7018-49a2-ac2d-b42321666f71") pod "control-p4tdq" (UID: "5db708ba-e3e5-4aab-85ea-919c61a9117e") "

@hujogrom determine if this is occurring on a specific set of nodes. You can attempt to manually mount from the Node where the deployment fails to mounts. Also, check your export policy for the backend where you are trying to mount.
If none of these options work then you should contact NetApp support to help you with problem determination.

exportPolicy is not relevant for iSCSI (only for NFS). In the case of iSCSI, access control is via the so-called iGroup, which you have configured in your backend config. And if the failure is not on specific nodes, it appears that the igroup is correct (it should contain the IQN of every k8s node). My guess would be a timeout issue in the iSCSI attach operation, though that should show up in the logs of the Linux host. Maybe worth a try to review the timeout values in /etc/iscsi/iscsid.conf as well as the multipath config if you use that.

Hi @wonderland, thanks a lot for your advice! I checked the settings you mentioned and see nothing suspicious, but maybe I'm missing something. Here are mine multipath.conf and iscsid.conf:
iscsid.conf:
iscsid.startup = /bin/systemctl start iscsid.socket iscsiuio.socket
iscsid.safe_logout = Yes
node.startup = automatic
node.leading_login = No
node.session.timeo.replacement_timeout = 120
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.initial_login_retry_max = 8
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.xmit_thread_priority = -20
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.conn[0].iscsi.HeaderDigest = None
node.session.nr_sessions = 1
node.session.iscsi.FastAbort = Yes
node.session.scan = auto
multipath.conf:
defaults {
user_friendly_names  no
max_fds              max
queue_without_daemon no
flush_on_last_del    yes
find_multipaths yes
disable_changed_wwids yes
verbosity 4
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]"
devnode "^(hd|xvd|vd)[a-z]"
devnode "^cciss!c[0-9]d[0-9][p[0-9]]"
}
Make sure our multipath devices are enabled.
blacklist_exceptions {
device {
vendor "NETAPP"
product "LUN"
}
}
devices {
device {
vendor                 "NETAPP"
product                "LUN"
features               "3 queue_if_no_path pg_init_retries 50"
hardware_handler       "1 alua"
prio                 "alua"
path_grouping_policy   group_by_prio
path_selector         "round-robin 0"
failback               immediate
rr_weight              uniform
rr_min_io              128
path_checker           tur
}
}

@hujogrom, we're not able to determine the likely cause of the problem from what you have reported so far. Please contact NetApp support so that we can collect some additional data.

Here are some suggestions to try:

The iscsi-initiator-utils package version must be <IP_ADDRESS>4-2.el7 or newer
Set node.session.timeo.replacement_timeout = 5 per 'NetApp ONTAP Documentation:
Setting the iSCSI replacement timeout value'
http://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.exp-iscsi-rhel-cg%2FGUID-F9935B47-A709-4CAA-945B-64C03E338E79.html
Include the following line in /etc/iscsi/iscsid.conf:
node.session.scan = manual

@hujogrom, please review this GitHub issue as it may be relevant to the issue you are seeing. Let us know if you've resolved the issue.

Closing this issue as the OP hasn't responded since 05/29/20.


add convenience for beach

Added functionality to make simulating beach matches much easier: removed explicit hard-coding of 4 sets to 25 + 1 set to 15 and added a convenience function vs_simulate_match_beach to include default parameters for 3-sets-to-21 beach volleyball.

Looks great. In retrospect my choice of serving5 as a parameter name was a poor one. I think we can soft-deprecate that and add serving_last and go_to_last (or similar) aliases - but not an immediate priority.


disallows underflows

using $n !== PHP_INT_MAX doesn't disallow underflows. this fixes the problem
by using bccomp to disallow underflows (since we want to allow PHP_INT_MIN).

Thanks! Tweaked slightly! https://github.com/cs50/php/commit/95e8f5d9bca21f9ffa11b7a138ad69a83797a1be


prettify on save doesn't respect preference

Expected behavior: File is not prettified.
Actual behavior: File is prettified
Steps to reproduce:

Check "Prettify On Save JSON" from the pretty-json settings pane.
Uncheck "Prettify On Save JSON" from the pretty-json settings pane.
Open a json that file that is not prettified.
Edit the file.
Save the file. Observe that it is now prettified on both the screen and the disk.

Reproducable: 100%

Did you find a fix @jonathankoren?

I completely misdiagnosed it. It turned out it, that Nuclide was overriding the preferences, because language-json was installed.
Next time

On May 30, 2018, at 9:52 AM, Amy Troschinetz<EMAIL_ADDRESS>wrote:
Did you find a fix @jonathankoren?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


Revert "Add and use secure_headers gem with default settings"

Reverts davidrunger/david_runger#238

I'm reverting this because:

I have realized that it doesn't work with prerendering (since the nonces don't match); this could maybe be fixed, but I don't think it's worth the effort/complexity
this breaks the flipper web UI and causes at least a warning in the sidekiq web UI


Modified edit blog form to look like other forms on the website

What is the change?
Made edit blog form to look similar to compose/ edit profile form
Related issue?
close: #950
How was it tested?
Tested Locally
Checklist:
Before you create this PR, confirm all the requirements listed below by checking the checkboxes [x]:

[x] Have you followed the Contribution Guidelines while contributing.
[x] Have you checked there aren't other open Pull Requests for the same update/change?
[x] Did you lint your code before making the Pull Request?
[] Have you made corresponding changes to the documentation?
[x] Your submission doesn't break any existing feature.
[x] Have you tested the code before submission?

Screenshots or Video:

Something went wrong here, opening new PR


sync with socket.io 1.4.3 - support compression

Please could you add the compression support, introduced in the latest socket.io?

WebSocketServerCompressionHandler should be used for this feature implementation. But it only available since netty-4.1.0.Beta8 version.

I don't see any problems with socket.io 1.4.4 client. Could you please clarify the problem?

I haven't tried this yet. Will be back in a few days

please try it with version master branch

Sorry, I haven't got how it should work.
In your class WebSocketTransport you do create default handshakers (see method handshake(....)), which deals nothing with PerMessageDeflateServerExtensionHandshaker

After some investigations, I suspect I have to insert the WebSocketServerCompressionHandler into the pipeline... But the exact position is not clear

Don't hurry with it. There is a bug https://github.com/netty/netty/issues/4754

Looks like I was able to get it work:

Just for curiosity, is it possible to get the compression statistics from the netty?

It's work because you Connection request header has Upgrade value only. You are lucky :)

compression statistics

What do you mean under statistics?

I'm using socket.io 1.4.4 client....
statistics - the total compression ratio. For now I do not see if the compression actually works - "frames" tab in Chrome dev tools shows just regular socket.io text frames

Confirm it works, but only with Chrome browser.

For now I do not see if the compression actually works

I tried it with javascript socketio server same situation - Chrome doesn't show any info about data compression. But socketio server shows compression flag turned on during data sending from server to client.

I can't get to work io.netty.handler.codec.http.HttpContentCompressor handler. Socketio client drops connection when this handler encodes the message.

@evgeny-pasynkov done

Have not found any compression statistics accumulation in netty code.

@evgeny-pasynkov  with netty 4.1.0CR2 it works both in firefox and chrome


Update to<EMAIL_ADDRESS>

Description
Updates the SDK to use the Apache 2.0 Licensed version of the Crypto Lib, located here

For contributor use:

[x] Targeted PR against master branch
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[x] Code follows the standards mentioned here.
[ ] Updated relevant documentation
[ ] Re-reviewed Files changed in the Github PR explorer
[ ] Added appropriate labels

Codecov Report

Merging #418 (6af3081) into master (ba0b0e0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #418   +/-   ##
=======================================
  Coverage   62.30%   62.30%
=======================================
  Files          24       24
  Lines        2966     2966
=======================================
  Hits         1848     1848
  Misses        994      994
  Partials      124      124

Flag
Coverage Δ

unittests
62.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

crypto/crypto.go
27.18% <ø> (ø)

crypto/hash.go
0.00% <ø> (ø)

event.go
0.00% <ø> (ø)

I'm on CODEOWNERS: https://github.com/onflow/flow-go-sdk/blob/master/CODEOWNERS
but merging is still bloicked 🤔


Documentation is not properly indexed by search engines

Take this page for example: https://turing.ml/dev/tutorials/02-logistic-regression/
Google and Bing are unable to find the page when searching either the title, quotes of entire paragraphs, or the url itself:
https://www.google.com/search?q=Turing.jl Bayesian Logistic Regression
https://www.google.com/search?q="In our example%2C we'll be working to predict whether someone is likely to default with a synthetic dataset found in the RDatasets package. This dataset%2C Defaults%2C comes from R's ISLR package and contains information on borrowers."
https://www.google.com/search?q=https%3A//turing.ml/dev/tutorials/02-logistic-regression/
Bing gives a more interesting result for the last one

https://cc.bingj.com/cache.aspx?q=https%3A%2F%2Fturing.ml%2Fdev%2Ftutorials%2F02-logistic-regression%2F&d=4682790180358358&mkt=en-US&setlang=en-US&w=FWl1uc863KZRfKX_yyAY6Evm1kh-V-1z
EDIT: Possibly related? https://github.com/TuringLang/Turing.jl/issues/1647

I don't have enough web-fu to understand this perfectly, but I wonder if it's because our robots.txt file points to a sitemap that does not seem to include the tutorials:
https://github.com/TuringLang/turing.ml/blob/gh-pages/dev/robots.txt
https://github.com/TuringLang/turing.ml/blob/gh-pages/dev/sitemap.xml

It seems work fine with keyword "Bayesian Logistic Regression Turing"


MWPW-134799: MEP On/Off

to be merged after https://github.com/adobecom/milo/pull/1531
MEP On/Off
This PR introduces a client-side on/off feature for MEP manifests.
Related Discussion
More details below.
Resolves:

MWPW-134799
MWPW-137499

Test URLs:

Before: https://main--milo--adobecom.hlx.page/products/special-offers?martech=off
After: https://onoff--milo--adobecom.hlx.page/products/special-offers?martech=off

Details
Promotions.xlsx
Promotions.xlsx - a separate global metadata file, configured in .helix/config.xlsx

located in root (milo folder)
holds connection folder/page-manifest and on/off date times.
serves metadata from the 'helix-default' tab
has the 'schedule' tab to configure on/off dates
The 'helix-default' has a hidden column called 'scheduled'. It is auto-generated from the 'schedule' tab and only visible when you preview promotions.xlsx.

Why a separate metadata file?

avoid conflicts & risks when GWP is modifying the main metadata file to configure promotions
easier overview & troubleshooting for promo issues
potential automation data file

Schedule Tab

no row/column can be deleted -> that would break generation functions in the 'concut' tab
datepicker & formatting for date fields
time formating for time fields
active/inactive controls if the schedule line is applied to metadata. GWP can set a line to inactive if they want to re-enable it later.

Priority
Promotions.xlsx is higher priority than metdata.xlsx
Promotions.xlsx only allows 2 fields to be configured, 'personalization' and 'events'. Its protected, so no other column from global metadata can be overwritten. On the contrary, global metadata should not be able to overwrite the 'personalization' and 'events' data in promotions.xlsx.
Priority is achieved due to order in .helix/config.xlsx
'manifestnames' and 'schedule' meta tags

'manifestnames' contains manifests attached to this particular page (similar to 'personalization' field, but instead of path GWP can use a so-called manifest nickname, from the schedule tab)
'schedule' contains all active promo schedule lines
'schedule' line format: manifest name | start datetime (GMT) | end datetime (GMT) | manifest path
Sample page

<meta name="manifestnames" content="pre-black-friday-global,black-friday-global,cyber-monday">
<meta name="schedule" content="pre-black-friday-global | 2023-11-01T00:00:00 | 2023-12-15T00:00:00 | https://main--milo--adobecom.hlx.page/promos/2023/black-friday/pre-black-friday/manifest-global.json, black-friday-global | 2023-12-15T00:00:00 | 2023-12-31T00:00:00 | https://main--milo--adobecom.hlx.page/promos/2023/black-friday/black-friday/manifest-global.json">

MEP behavior

MEP will read 'schedule' metadata
MEP will sort out of 'schedule' promo manifests attached to this page
For each manifest it will decide if its enabled (scheduled for current time) or disabled (scheduled outside of current time or the schedule is missing at all)
MEP will handle the manifests from 'personalization' and the ones from 'promo' in the same way. The 'personalization' manifests are lower priority - to be discussed.
if 'bts' event is found, the start and end datetime are converted to a local browser time, then compared with the current time. If it's on - manifest is enabled, if not - disabled.
disabled manifest is NOT preloaded and NOT fetched. This is true for both hlx.page and hlx.live.
Preview for promo manifests work the same as for usual personalization.

Preview
'Scheduled - inactive/active' shows if the current browser time is within the manifest's on/off period.
On and Off date time is shown in the local timezone, NOT in GMT.
When there is a manifest overwrite in the URL, MEP will disregard the schedule. This is needed for QA-ing inactive changes.

for now rebased on mwpw-138750-analytics2 as agreed with @vivgoodrich,
once pull/1531 is merged, this PR will be rebased to the main and ready to merge.


Ultra PSET Spacecraft Frame + PSET Helio Frame Exposure Times

Change Summary
Overview
Finished pset spacecraft frame EXCEPT background rates
Added pset helio frame exposure times
New Files

dps_sensitivity45.cdf

Lookup table

Updated Files

ultra_l1c_pset_bins.py

Finished pset spacecraft frame EXCEPT background rates
Added pset helio frame exposure times

conftest.py

Modified path

constants.py

Added constants

Testing

test_ultra_l1c_pset_bins.py

Tests ultra_l1c_pset_bins.py

Test data added: dps_exposure_helio_45_E1.cdf, dps_exposure_helio_45_E12.cdf, dps_exposure_helio_45_E24.cdf

pre-commit.ci autofix

There may be a problem merging this back into the dev branch because of unsynchronized changes to the decom.py file. In the PR branch, line 11 of this file is:
from space_packet_parser import parser, xtcedef
whereas on the dev branch it is
from space_packet_parser import definitions
The dev branch code is more recent and is part of a more comprehensive change that upgrades  space_packet_parser from version 4 to version 5. It looks like this PR was branched before these changes were made to dev.
The bottom line here is that the code in this PR will not run with version 5 of the space_packet_parser, i.e., this code won't run with the package configuration specified by the poetry.lock file currently in dev.
This may be as simple as changing decom.py to the new version in dev, but I really have no idea what will happen if you try to merge it back in.


generator: upgrade cryptography package version in poetry.lock

This should solve a few security alerts, for example https://github.com/chairemobilite/evolution/security/dependabot/73

@samuel-duhaime I don't know how one sets the version of the packages in python. If, like nodejs, you just need to upgrade the version of one package then run something? Or do some other command to upgrade?

There's also a few other python packages that have required updates to solve security issues. See the Security tab and dependabot alerts.

Okay, I will try to have a look at this.


refactor(frontend): rearrange column binding steps in create table handler

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?

Checklist

[ ] I have written necessary rustdoc comments.
[ ]  I have added necessary unit tests and integration tests.
[ ]  I have added test labels as necessary.
[ ]  I have added fuzzing tests or opened an issue to track them.
[ ]  My PR contains breaking changes.
[ ]  My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
[ ]  My PR contains critical fixes that are necessary to be merged into the latest release.

Documentation

[ ]  My PR needs documentation updates.

Release note

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#19917  👈 (View in Graphite)
#19916
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Improve testimonial

Css  and js codes are not working properly in testimonial  page .So I want to fix that and make the ui better.Kindly assign this issue to me under JWOC(phase 2)

I would like to work on this under JWOC


chore: update lockfile

This updates the lockfile, which we should do periodically.
A lot of file changed as the snapshots hashes updated (would be great to not have hashes in these tests at some point)
I had to update the the rollup-plugin-index-html test to handle new rollup syntax.

@daKmoR did you merge this as part of the other merge?

no I did not... probably needs to be regenerated as some versions of demoing-storybook changed 🙈


Use align-deps to help migrate apps to the New Architecture

For app maintainers to migrate to the New Architecture, all of their native dependencies must support the New Architecture as well. This isn't the same as being merely compatible with some React Native version 0.x, since (for the time being) the New Architecture is opt-in.
A problem that has come up is that it's not easy to know ahead of time, as an app maintainer, whether all of your dependencies have new versions available that are compatible with the New Architecture. In the worst case, you might sink time and effort into beginning an upgrade, only to get blocked halfway by some third-party library and have to abandon the effort for an unknown length of time.
The idea here would be to add information to align-deps's database about various common libraries' New Architecture readiness, such that (ideally) we can recommend align-deps as part of the official New Architecture migration guide for app maintainers. In doing this, we want to (1) increase the likelihood of apps migrating successfully on the first try, (2) create visibility in the ecosystem (and thus momentum and excitement) around popular libraries migrating to the New Architecture.
There are some questions around how we source this information reliably at scale (e.g. combine efforts with https://reactnative.directory/ ?), but in the short term leveraging align-deps's standalone database of packages seems like a good way to jumpstart this capability for relatively low effort.
NOTE: I'm only lightly familiar with align-deps and the New Architecture, so I've kept this super high-level and am not opinionated on the details.
cc @cortinico @tido64 @afoxman @huntie @cipolleschi

update by @kelset: changed dep-check to align-deps ;)

We can probably use the list in https://github.com/reactwg/react-native-new-architecture/discussions/6 as a starting point.

We can probably use the list in reactwg/react-native-new-architecture#6 as a starting point.

I have a draft PR with only these packages. The rest are null-ed out.

I think this would be an amazing addition to the New Architecture migration story.
It has been asked previously by the community (see https://github.com/facebook/react-native-website/issues/3275).
As for the technical side: we don't have a simple way to know if a library is New Architecture compatible. In general, relying on codegenConfig inside package.json is a good indicator. However there might be other libraries which are New Architecture compatible and not using the Codegen. We should think about an indicator that allows us to easily understand this (like another field in the package.json or so).

Is codegenConfig always present in a fully migrated package? I extended the script we use to check for updated packages to include it:

Capability
Name
Version
Latest
New Arch
Homepage

animation
react-native-reanimated
^2.10.0
=

https://github.com/software-mansion/react-native-reanimated#readme

base64
react-native-base64
^0.2.1
=

https://github.com/eranbo/react-native-base64#readme

checkbox
@react-native-community/checkbox
^0.5.8
0.5.12

https://github.com/react-native-community/react-native-checkbox#readme

clipboard
@react-native-clipboard/clipboard
^1.10.0
1.11.0

https://github.com/react-native-clipboard/clipboard#readme

datetime-picker
@react-native-community/datetimepicker
^6.0.2
6.3.2

https://github.com/react-native-community/datetimepicker#readme

filesystem
react-native-fs
^2.18.0
2.20.0

https://github.com/itinance/react-native-fs#readme

floating-action
react-native-floating-action
^1.22.0
=

https://github.com/santomegonzalo/react-native-floating-action#readme

gestures
react-native-gesture-handler
^2.6.0
=
Yes
https://github.com/software-mansion/react-native-gesture-handler#readme

hooks
@react-native-community/hooks
^2.8.0
2.8.1

https://github.com/react-native-community/hooks#readme

html
react-native-render-html
^6.1.0
6.3.4

https://meliorence.github.io/react-native-render-html/

jest
jest
^26.6.3
29.0.2

https://jestjs.io/

lazy-index
@rnx-kit/react-native-lazy-index
^2.1.7
=

https://github.com/microsoft/rnx-kit/tree/main/packages/react-native-lazy-index#readme

masked-view
@react-native-masked-view/masked-view
^0.2.7
=

https://github.com/react-native-masked-view/masked-view#readme

modal
react-native-modal
^13.0.0
13.0.1

https://github.com/react-native-community/react-native-modal

navigation/native
@react-navigation/native
^6.0.8
6.0.12

https://reactnavigation.org

navigation/stack
@react-navigation/stack
^6.2.0
6.2.3

https://reactnavigation.org/docs/stack-navigator/

netinfo
@react-native-community/netinfo
^9.0.0
9.3.0

https://github.com/react-native-netinfo/react-native-netinfo#readme

popover
react-native-popover-view
^5.0.0
5.1.2

https://github.com/steffeydev/react-native-popover-view#readme

safe-area
react-native-safe-area-context
^4.3.1
4.3.3
Yes
https://github.com/th3rdwave/react-native-safe-area-context#readme

screens
react-native-screens
^3.14.1
3.17.0
Yes
https://github.com/software-mansion/react-native-screens#readme

shimmer
react-native-shimmer
^0.5.0
0.6.0

https://github.com/oblador/react-native-shimmer

sqlite
react-native-sqlite-storage
^6.0.1
=

https://github.com/andpor/react-native-sqlite-storage

storage
@react-native-async-storage/async-storage
^1.17.10
=

https://github.com/react-native-async-storage/async-storage#readme

svg
react-native-svg
^12.3.0
13.1.0
Yes
https://github.com/react-native-community/react-native-svg

test-app
react-native-test-app
^1.6.9
1.6.13

https://github.com/microsoft/react-native-test-app

webview
react-native-webview
^11.23.0
11.23.1

https://github.com/react-native-webview/react-native-webview#readme

Reanimated isn't listed here because New Arch support is still in RC.

Is codegenConfig always present in a fully migrated package? I extended the script we use to check for updated packages to include it:

As mentioned above, not always. There might be packages without codegenConfig which support the New Architecture. That being said, I think it's anyway a good indicator.

This conversation I think is very close to what the RNDirectory is trying to achieve through this: https://github.com/react-native-community/directory/pull/870
cc @simek
It'd be great to decide a standard way to communicate new arch support, and then "enforce it" by having both dep-check and directory leverage it as the official way. At that point we can communicate in all the 3 places (dep-check, RN docs website, RNDirectory) that doing "XXX" is the one way.

I've been wondering if it makes more sense for each package to add a field that signals that they are new-arch ready. It would allow cli to warn when autolinking, dep-check/directory to generate the compat table, and docs to be automatically updated.

I've been wondering if it makes more sense for each package to add a field that signals their new-arch readiness.

Yup that's a good thing to add. Do you have any suggestion (I assume it's going to be stored inside the package.json or the react-native.config.js in some form).

Yup that's a good thing to add. Do you have any suggestion (I assume it's going to be stored inside the package.json or the react-native.config.js in some form).

I think package.json would make the most sense since you can query it using npm instead of downloading the full package just to inspect a single file.

@cortinico @tido64 Small input on using package.json:
For an upcoming feature in Metro (exports support), we:

Preliminarily aligned on package.json over react-native.config.js as the location for package-defined options.
Are considering a root react-native key to encapsulate these options.

Therefore if we are looking to introduce new explicit key(s) for this feature, it might sit well under react-native:
{
  "name": "my-package",
  "react-native": {
    "metro-exports-mode": "strict",
    "new-arch-support": "ready"
  }
}

@huntie We can't use react-native, specifically - that key is already used as a React Native-specific variant of browser.

@huntie: Yes, I think adding a new field like you described makes the most sense. It would allow for future expansions.
If we're all agreeing on this approach, what would be the next steps? In my mind, I'm thinking we should at least start with:

[ ] Communicating this "requirement" to the community (Meta?)
[ ] Update rn-cli to scan for this field if new arch is enabled, and warn about potentially unsupported modules and teach users how to get rid of the warning (Callstack?)
[ ] Add the dep-check profile with new arch only packages to help users upgrade their projects, periodically updated using a script that scans this new field (Microsoft)
[ ] rn-directory to update their scripts (Expo?)

Unrelated: Would it also make sense to move codegenConfig into react-native-config?

cc @brentvatne in case you have better ideas or feedback on this.

Another interesting bit I discovered today: Even with codegenConfig present, it's not a guarantee that the library supports New Arch + autolinking. I've spent days (on separate occasions) trying to get to the bottom of this:  https://github.com/th3rdwave/react-native-safe-area-context/issues/303
Which brings me back to this:

If we're all agreeing on this approach, what would be the next steps? In my mind, I'm thinking we should at least start with:

[ ]  Communicating this "requirement" to the community (Meta?)
[ ]  Update rn-cli to scan for this field if new arch is enabled, and warn about potentially unsupported modules and teach users how to get rid of the warning (Callstack?)
[ ]  Add the dep-check profile with new arch only packages to help users upgrade their projects, periodically updated using a script that scans this new field (Microsoft)
[ ]  rn-directory to update their scripts (Expo?)

Unrelated: Would it also make sense to move codegenConfig into react-native-config?

I'd love to hear what people think should be the way forward.

Even with codegenConfig present, it's not a guarantee that the library supports New Arch + autolinking

That's what I mentioned before. We need an explicit field to mention that the New Architecture is supported.
Is this the preferred approach then? nit: react-native-config vs codegenConfig are mixing camel case with kebab case, not great.
  "react-native-config": {
    "metro-exports-mode": "strict",
    "new-arch-support": "ready"
  }

Unrelated: Would it also make sense to move codegenConfig into react-native-config?

Potentially yes. Yet please note that both Gradle and CocoaPods are parsing the package.json. Moving things inside a react-native.config.js which is not a JSON file, will make it hard to parse it.
Also speaking with @cipolleschi, we were brainstorming about having a single variable "new-arch-enabled": "true" for Applications to enable/disable the new arch (instead of having separate configs for iOS and Android). If we do so, it would be great to use the same key/value for Libraries and Apps.

Is this the preferred approach then? nit: react-native-config vs codegenConfig are mixing camel case with kebab case, not great.
  "react-native-config": {
    "metro-exports-mode": "strict",
    "new-arch-support": "ready"
  }

I'm not a fan of mixing cases either. We should probably use camelCase (i.e. reactNativeConfig) since that seems to be the style used in package.json

Unrelated: Would it also make sense to move codegenConfig into react-native-config?

Potentially yes. Yet please note that both Gradle and CocoaPods are parsing the package.json. Moving things inside a react-native.config.js which is not a JSON file, will make it hard to parse it.

I meant the react-native-config key (or reactNativeConfig), not the file. Sorry for the confusion 😛

gentle nudge to the thread: @huntie how is the work on the export feature? Have you decided on the package.json field name? react-native-config or reactNativeConfig?

@kelset Very close to publishing, will include the "exports" RFC link here when done: [incoming].
Standardising on camel case throughout makes sense. The RFC will contain a new proposed key for package.json, "reactNative":
  "reactNative": {
    "metroExportsMode": "strict",
    "newArchSupport": "ready"
  }

With this casing change, we can end up keeping a simple and wide name to group these React Native config options and/or future metadata. The downside here is the chance of confusion with existing key.

I guess we can then bring over the conversation about the name of the config to the RFC once it's up. Not a big fun of having "react-native" and "reactNative" be two totally different things.

Hey folks; just wanted to follow up here now that thanks to @huntie and the rest of the Metro team it seems that the exports support is going ahead and the definition of "react-native" has been accepted in node itself (https://github.com/nodejs/node/pull/45367).
If I'm understanding correctly, it means that "react-native" is now officially used for that - and now we need to create a different name for section in the package.json along the lines of:
  "react-native-config": {
      "custom-setting": "entry",
      ...
  },
  "dependencies": {
     // etc etc

Right?
And not only that, we need to decide what to name the field for identifying react-native's new arch support, right?

On top of this, I wanted to also check in into an idea that the developers from InfiniteRed brought to us for indicating via align-deps which libs support the new arch or not.
Basically, they were proposing something a bit more in-depth than just a binary yes/no, but something more like a semaphore:

🔴 = no support for new arch at all
🟡 = we can infer that it will probably work with the new arch based on . For example, we could use the same one that RNDirectory uses, based on the presence of the codegenConfig field in package.json
🟢 = library has our custom-setting field set to true

What do you think? I think this might help and also we could potentially "mirror" this into the RNDirectory website too.

it means that "react-native" is now officially used for that

Clarifying: The "react-native" root field was originally defined and used by us (Metro, Webpack, others) as an alternative for the "browser" field spec/"main" field, and is already in wide use (with package exports this concept is being replaced — however we will not be pushing packages to migrate and remove root field).
The impact is the same, however! We need a different root field name to "react-native" to avoid conflict, such as"react-native-config" (as previously suggested) or "reactNative" (potentially too indistinct).

Totally, thanks for the clarification @huntie - we did a meeting earlier this week with @motiz88 and Blake and other Meta folks, I'm guessing the next step is to understand how we want to go ahead with this. My current latest understanding is that internally during planning you will decide how you want to go ahead with this in H123 and then we can create some actionables.

Reviving this thread one more time, since now we should all be approaching clarity over how the next few months would look like.
I feel that the next step would be to get a few folks in a room and make sure that we can reach an agreement on the final "spec" for this new react-native-config section of the package.json. I'm a bit foggy but I am pretty sure that Expo and the RN CLI already have some variations of local configurations (IIRC, using app.json and react-native.config.js) so I reckon that whatever shape we end up agreeing on, we should realign all these "side configurations" into a main, unified and spec'd one.
I'd like to get a few folks in a room to discuss details and after that, setup a "final" RFC for how to approach so that we can coordinate the needed work. As attendees, I'm thinking some representatives from CLI, Expo, Metro and rnx-kit, keeping the number of people below 10.

quick update: we are now working on said configuration standard, read more here: https://github.com/react-native-community/discussions-and-proposals/pull/480

headsup: we are still blocked on this waiting on the RFC to make progress: https://github.com/react-native-community/discussions-and-proposals/pull/588
AFAIK it will be actually get closed in favour of a 3.0 rewrite as a separate PR; I'll post an update once it's there.

An update on this: since the last time we talked about this, there have been a couple of relevant updates:

libraries have started supporting new arch & bridgeless. So, indirectly, just by keeping up and having profiles for 0.73 and 0.74 for align-deps, we do have somewhat a level of support for developers using align-deps in making sure that they are using versions of libs that do have newarch/bridgeless support.

@aleqsio made a small handy tool that checks a repository package.json against the (manually maintained) gSheet linked in the discussion above. While not ideal nor programmatic (since it's source of truth is a man-mantained gSheet), me and Tommy think it does a pretty great job.

So devs at this point in time have a pretty decent level of support in migrating their apps to the new arch.
Because of this, we think we can close this off. Let us know if you have other thoughts on the matter.