[WIP][SPARK-29672][PYSPARK] remove py27 support for jenkins tests

What changes were proposed in this pull request?
remove python2.7 tests and test infra for 3.0+
Why are the changes needed?
because python2.7 is finally going the way of the dodo.
Does this PR introduce any user-facing change?
newp.
How was this patch tested?
the build system will test this

Test build #112965 has started for PR 26330 at commit 92f7268.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17888/
Test PASSed.

Test build #112966 has started for PR 26330 at commit 7c0fe46.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17889/
Test PASSed.

Test build #112965 has finished for PR 26330 at commit 92f7268.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112965/
Test PASSed.

Test build #112966 has finished for PR 26330 at commit 7c0fe46.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112966/
Test PASSed.

Thank you for making this PR, @shaneknapp .
From the Jenkins log, PIP installation test seems to use Python 3.5 still. Could you take a look at that?
Testing pip installation with python 3.5
Using /tmp/tmp.MBOPq46hzq for virtualenv

Thank you for making this PR, @shaneknapp .
From the Jenkins log, PIP installation test seems to use Python 3.5 still. Could you take a look at that?
Testing pip installation with python 3.5
Using /tmp/tmp.MBOPq46hzq for virtualenv

that's actually in dev/run-pip-tests...  i guess i can hack on that as it's kinda in the scope of this PR.

Test build #112968 has started for PR 26330 at commit 226c1fa.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17892/
Test PASSed.

that's actually in dev/run-pip-tests... i guess i can hack on that as it's kinda in the scope of this PR.

btw i've never really looked too closely at run-pip-tests, so i may have a few extra commits as i mess around w/it trying to get it to use 3.6, and strip out the python2 stuff.

Test build #112972 has started for PR 26330 at commit f25c4b3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17895/
Test PASSed.

Test build #112972 has finished for PR 26330 at commit f25c4b3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112972/
Test FAILed.

Test build #112968 has finished for PR 26330 at commit 226c1fa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112968/
Test PASSed.

test this please

Test build #112977 has started for PR 26330 at commit f25c4b3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17900/
Test PASSed.

Test build #112977 has finished for PR 26330 at commit f25c4b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112977/
Test PASSed.

okie dokie, i'm confident that this change won't break anything, but i'll wait on merging until i get the word.
@HyukjinKwon @mengxr @srowen

Seems OK to me, though I don't know this script well.

this is @holdenk 's contribution...  should have added her to the review earlier.
also, if you look at the console output (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112977/console), the following block of code in dev/run-pip-tests causes some really ugly log4j errors that i would LOVE to clean up (but i'm not entirely sure if that's easy/possible):
    echo "Run basic sanity check on pip installed version with spark-submit"
    spark-submit "$FWDIR"/dev/pip-sanity-check.py
    echo "Run basic sanity check with import based"
    python "$FWDIR"/dev/pip-sanity-check.py
    echo "Run the tests for context.py"
    python "$FWDIR"/python/pyspark/context.py

Run basic sanity check on pip installed version with spark-submit
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: target/unit-tests.log (No such file or directory)
	at java.io.FileOutputStream.open0(Native Method)
<SNIP>
Successfully ran pip sanity check

Test build #113043 has started for PR 26330 at commit b57acd9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17964/
Test PASSed.

The code change LGTM provided it runs in Jenkins
As for the log4j errors clean up I think we can address that in a follow up. My guess is we need to set an env var and create a file but I think it's ok to move forward with this first.

Test build #113043 has finished for PR 26330 at commit b57acd9.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113043/
Test FAILed.

test this please

Test build #113050 has started for PR 26330 at commit b57acd9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17971/
Test PASSed.

Test build #113050 has finished for PR 26330 at commit b57acd9.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113050/
Test FAILed.

Test build #113050 has finished for PR 26330 at commit b57acd9.

This patch fails PySpark unit tests.

i'll revert my pypy3 change and let the test run again...  this is officially out of scope of this pr.  :)

Test build #113059 has started for PR 26330 at commit 0b37b71.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17980/
Test PASSed.

Actually, @shaneknapp, I think we should remove pypy 2.5 too, which is compatible with Python 2 ... does it make things complicated?

Ah, tests fails with:
py4j.protocol.Py4JError: org.apache.spark.SparkConf.__name__ does not exist in the JVM

Okay, this one I think I can fix it in a separate PR. I will make a followup after this PR.

Actually, @shaneknapp, I think we should remove pypy 2.5 too, which is compatible with Python 2 ... does it make things complicated?

nope, it makes things easier.  :)

Test build #113059 has finished for PR 26330 at commit 0b37b71.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113059/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18020/
Test PASSed.

Test build #113104 has started for PR 26330 at commit be57b53.

Test build #113104 has finished for PR 26330 at commit be57b53.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113104/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18021/
Test PASSed.

Test build #113106 has started for PR 26330 at commit 074947b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18023/
Test PASSed.

Test build #113106 has finished for PR 26330 at commit 074947b.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113106/
Test FAILed.

Test build #113106 has finished for PR 26330 at commit 074947b.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dafuq?

test this please

Test build #113108 has started for PR 26330 at commit 074947b.

Merged build finished. Test PASSed.

Test build #113108 has finished for PR 26330 at commit 074947b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113108/
Test FAILed.

test this please

Test build #113114 has started for PR 26330 at commit 074947b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18028/
Test PASSed.

Test build #113114 has finished for PR 26330 at commit 074947b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113114/
Test PASSed.

ok, i'm back from being out sick and will address these comments and whatnot today.
i've also looked at the other test framework and will be submitting additional changes...

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113282/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18190/
Test PASSed.

test this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18191/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113283/
Test FAILed.

Test FAILed.

that was me, i killed the job.
i spent some time this morning looking deeper in to how we call these myriad tests, how we determine which python executable to run, etc etc and i just pushed a couple of commits that i think make things happen in a more understandable way.

Test build #113286 has started for PR 26330 at commit cab1683.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18194/
Test PASSed.

talk about unintended consequences:  this PR is breaking the pretty-printed output on jenkins:
java.lang.Thread.run(Thread.java:748)\n'b'13:42:45.815 WARN org.apache.spark.sql.execution.command.DropTableCommand: org.apache.spark.sql.AnalysisException: Table or view not found: view9; line 1 pos 14;\n'b"'SubqueryAlias `default`.`view10`\n"b'+- View (`default`.`view10`, [id#431310L,id1#431311L])\n'b"   +- 'Project [*]\n"b"      +- 'UnresolvedRelation [view9]\n"b'\n'b'org.apache.spark.sql.AnalysisException: Table or view not found: view9; line 1 pos 14;\n'b"'SubqueryAlias `default`.`view10`\n"b'+- View (`default`.`view10`, [id#431310L,id1#431311L])\n'b"   +- 'Project [*]\n"b"      +- 'UnresolvedRelation [view9]\n"b'\n'b'\tat org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)\n'b'\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:95)\n'b'\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:90)\n'b'\tat org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:154)\n'b'\tat

wonder if this is from from __future__ import print_function...  guess i should pull them out as we won't need them anymore and see if that fixes it.

Test build #113286 has finished for PR 26330 at commit cab1683.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113286/
Test FAILed.

test this please

Test build #113399 has started for PR 26330 at commit cab1683.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18297/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113399/
Test FAILed.

Test build #113401 has started for PR 26330 at commit a5b056b.

Test build #113401 has finished for PR 26330 at commit a5b056b.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113401/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18299/
Test PASSed.

test this please

i just realized that if we're going to run everything under py3.6+, then we need to have the correct version of flake8 installed on the workers to deal w/this.  :)
upgrade/install done and hopefully my tests will run!

Test build #113402 has started for PR 26330 at commit a5b056b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18300/
Test PASSed.

i just realized that if we're going to run everything under py3.6+, then we need to have the correct version of flake8 installed on the workers to deal w/this. :)
upgrade/install done and hopefully my tests will run!

starting flake8 test...
flake8 checks passed.

talk about unintended consequences: this PR is breaking the pretty-printed output on jenkins:
java.lang.Thread.run(Thread.java:748)\n'b'13:42:45.815 WARN org.apache.spark.sql.execution.command.DropTableCommand: org.apache.spark.sql.AnalysisException: Table or view not found: view9; line 1 pos 14;\n'b"'SubqueryAlias `default`.`view10`\n"b'+- View (`default`.`view10`, [id#431310L,id1#431311L])\n'b"   +- 'Project [*]\n"b"      +- 'UnresolvedRelation [view9]\n"b'\n'b'org.apache.spark.sql.AnalysisException: Table or view not found: view9; line 1 pos 14;\n'b"'SubqueryAlias `default`.`view10`\n"b'+- View (`default`.`view10`, [id#431310L,id1#431311L])\n'b"   +- 'Project [*]\n"b"      +- 'UnresolvedRelation [view9]\n"b'\n'b'\tat org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)\n'b'\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:95)\n'b'\tat org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:90)\n'b'\tat org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:154)\n'b'\tat

wonder if this is from from __future__ import print_function... guess i should pull them out as we won't need them anymore and see if that fixes it.

ok, it looks like this is caused by the chain of dev/run-tests.py running with python3 calling sbt (for various build steps) through the run_cmd() function in dev/sparktestsupport/shellutils.py library.  none of the build steps have the return_output=True flag for run_cmd set, so the output is just dumped out by the subprocess itself as a python bytestring.
since we're not decoding the output, it just dumping those bytestrings to the build log.
removing from __future__ import print_function did seem to fix some of the output issues, but i'll have to think a little bit about better handling this.

Test build #113402 has finished for PR 26330 at commit a5b056b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113402/
Test FAILed.

Test build #113406 has started for PR 26330 at commit aa893c6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18304/
Test PASSed.

Test build #113406 has finished for PR 26330 at commit aa893c6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113406/
Test FAILed.

Test build #113407 has started for PR 26330 at commit 3036fa6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18305/
Test PASSed.

sigh.  :(
my attempt to fix the build output (https://github.com/apache/spark/pull/26330/commits/aa893c60b2af080d887468042c8a2c403b9d4100) doesn't work.  from the most recent build (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113407/), we're still getting output like this:
\x1b[0m\x1b[32mLabelPropagationSuite:\x1b[0m\x1b[0m\n'b'\x1b[0m[\x1b[0minfo\x1b[0m] \x1b[0m\x1b[32m- Watch events should be pushed to the snapshots store as snapshot updates. (27 milliseconds)\x1b[0m\x1b[0m\n'b'\x1b[0m[\x1b[0minfo\x1b[0m]

since the entire point of this PR is to get the testing infra (and tests) off of python2.7, i consider this to be close to a blocker if the build log output is completely illegible.
i'm open to any and all suggestions at this point.  :)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113407/
Test FAILed.

Test build #113408 has started for PR 26330 at commit 99a945b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18306/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113408/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18308/
Test PASSed.

Test build #113410 has started for PR 26330 at commit 4986769.

Test build #113410 has finished for PR 26330 at commit 4986769.

This patch fails executing the dev/run-tests script.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113410/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18311/
Test PASSed.

Test build #113413 has started for PR 26330 at commit 465acf1.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113413/
Test FAILed.

Test build #113414 has started for PR 26330 at commit b66b389.

Test build #113414 has finished for PR 26330 at commit b66b389.

This patch fails executing the dev/run-tests script.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113414/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18312/
Test PASSed.

Test build #113416 has started for PR 26330 at commit 64ccb81.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18313/
Test PASSed.

Test build #113416 has started for PR 26330 at commit 64ccb81.

ARGH.  closer...  at least SOME stuff is getting properly printed out.  :(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113416/
Test FAILed.

Test build #113419 has started for PR 26330 at commit 16ae236.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18316/
Test PASSed.

Test build #113426 has started for PR 26330 at commit 5b61ade.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18321/
Test PASSed.

Test build #113419 has finished for PR 26330 at commit 16ae236.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113419/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113431/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18326/
Test PASSed.

Test build #113433 has started for PR 26330 at commit 137371d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18327/
Test PASSed.

i'll do another pass on finding Popen calls and adding decode on monday to ensure few build breakages.

Test build #113426 has finished for PR 26330 at commit 5b61ade.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113426/
Test PASSed.

Test build #113433 has finished for PR 26330 at commit 137371d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113433/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18359/
Test PASSed.

Test build #113467 has started for PR 26330 at commit 48e318b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18362/
Test PASSed.

Test build #113470 has started for PR 26330 at commit a432afc.

Test build #113467 has finished for PR 26330 at commit 48e318b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113467/
Test PASSed.

Test build #113470 has finished for PR 26330 at commit a432afc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113470/
Test PASSed.

the deeper i look, the more stuff i see needs to be updated.  i'll be pushing some changes to dev/run-pip-tests that i missed earlier.

Test build #113645 has started for PR 26330 at commit 4ad08b0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18529/
Test PASSed.

Test build #113645 has finished for PR 26330 at commit 4ad08b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113645/
Test PASSed.

@srowen @HyukjinKwon @BryanCutler
i think this PR is good to go...  the implications, however, are pretty big:  no more test support for python2.7 for 3.x and up.
what are everyone's thoughts on this?  i sent something to dev@ last week but haven't gotten any responses yet.

Actually, my initial thought is that we drop everything right after Spark 3.0 branch is cut out (including upgrading R version to 3.4 as well to address intermittent CRAN check failure) which is strictly correct.
However, it's not always able to match the exact timing in practice and those infer/build stuff need all your efforts and time. So TBH I just don't want to say let's hold off at this moment until 3.0.
I will leave it to you and other committers here.

Actually, my initial thought is that we drop everything right after Spark 3.0 branch is cut out (including upgrading R version to 3.4 as well to address intermittent CRAN check failure) which is strictly correct.
However, it's not always able to match the exact timing in practice and those infer/build stuff need all your efforts and time. So TBH I just don't want to say let's hold off at this moment until 3.0.
I will leave it to you and other committers here.

imho it's better to get the test/build infra upgraded before new versions are cut...  if we still want to 'unofficially' support python2.7/pypy2.5.1 right up until we release 3.0, i can add those back to python/run-tests.py.
in fact, i will do that now "just to see what happens".  :)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18602/
Test PASSed.

Test build #113722 has started for PR 26330 at commit b5bada3.

My thought is that it's not that practical to test every PR against every Python version we support. We should focus on maybe 2 of the most widely used versions. Since we still need to support these versions, we can setup pre-release tasks to run the full tests and correct any issues that have come up. I think any issues will mostly be minor syntax problems and should be easily corrected.

My thought is that it's not that practical to test every PR against every Python version we support. We should focus on maybe 2 of the most widely used versions. Since we still need to support these versions, we can setup pre-release tasks to run the full tests and correct any issues that have come up. I think any issues will mostly be minor syntax problems and should be easily corrected.

so, which two versions would you suggest?  3.6?  2.7?  some other minor version of 3?

Test build #113722 has finished for PR 26330 at commit b5bada3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113722/
Test PASSed.

ok, i have an idea:
how about i change the scope of this PR (and JIRA) to solely update the test/build infrastructure to python3.  i will leave the python2.7/pypy2.5.1 test calls in place (see https://github.com/apache/spark/commit/b5bada3edaedb7b21d108e90047f5c9f77e6f6a9) for now.
once we cut the first 3.0 release, we can decide exactly which major/minor versions of python to test against on master and update python/run-tests.py accordingly.
sound reasonable?

so, which two versions would you suggest? 3.6? 2.7? some other minor version of 3?

If we are choosing 2 versions, I think 3.6 and maybe 3.8.

once we cut the first 3.0 release, we can decide exactly which major/minor versions of python to test against on master and update python/run-tests.py accordingly.

That sounds like a good plan if it's not too much hassle for you. I would get a little nervous about some Python 2 issue holding up the release :)

Seems OK to me if it's forward progress. But we drop it entirely after branch 3.0 is cut?

If we are choosing 2 versions, I think 3.6 and maybe 3.8.

that works...  it will take a little bit of work to get a 3.8 conda env set up, but i should be able to get that sorted by the end of january 2020.

once we cut the first 3.0 release, we can decide exactly which major/minor versions of python to test against on master and update python/run-tests.py accordingly.

That sounds like a good plan if it's not too much hassle for you. I would get a little nervous about some Python 2 issue holding up the release :)

well, tests are currently passing w/python2...  which is great.  if we see that python2 tests are going to hold up the 3.0 release, i can just yank the python2.7 (and pypy) executables from python/run-tests.py!

Seems OK to me if it's forward progress. But we drop it entirely after branch 3.0 is cut?

yeah, it's forward progress and i will be overjoyed to drop python2.7/pypy2.5.1 after 3.0.
i'll update the PR title/desc, as well as the JIRA and then merge later today.

thanks for the reviews everyone!  much appreciated...  :)

BTW, @shaneknapp quick question. did we change default python executable from Python 2 to Python 3? I was investigating some flaky tests (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113824/testReport/org.apache.spark.sql.streaming.continuous/ContinuousSuite/continuous_mode_with_various_UDFs___Scalar_Pandas_UDF/history/) which uses python executable and wondering if there's any delta recently.

Hi, @shaneknapp .
Could you update the Jenkins job scripts? Three Jenkins jobs seem to be broken.

https://amplab.cs.berkeley.edu/jenkins/view/Spark QA Test (Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/configure

- ZINC_PORT=$(python -S -c "import random; print random.randrange(3030,4030)")
+ ZINC_PORT=$(python -S -c "import random; print(random.randrange(3030,4030))")

Oh, NVM about my comment. It was because https://github.com/apache/spark/pull/26133

Hi, @shaneknapp .
Could you update the Jenkins job scripts? The following Jenkins job seems to be broken.

https://amplab.cs.berkeley.edu/jenkins/view/Spark QA Test (Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/configure

- ZINC_PORT=$(python -S -c "import random; print random.randrange(3030,4030)")
+ ZINC_PORT=$(python -S -c "import random; print(random.randrange(3030,4030))")

yeah, i missed a couple of those.  they're all fixed now.

Thank you, @shaneknapp !

Hey @HyukjinKwon , did you resolve the error
This progress looks :| because there were missing external dependencies ?
Could you please point me to changes if you did resolve it ?
I"m having the same error when trying to run PySpark using Python 3.

Can you file a JIRA with the reproducible steps?


paid theme on themeforest !

hi,
your work is awesome! there are very few readymade flask admin templates out there. FYI, if you put this on themefores - there will be a lot of us who are willing to pay for it.
thanks!

Hi @sandys
Just to be clear, I am not the author of gentelella, the front-end template that I'm using here. You can find the original template here: https://github.com/puikinsh/gentelella (there is also a link in the readme)
Gentelella is the front-end and what I did here is to add a Flask back-end, along with the whole CI/CD stuff (pytest, linting, selenium, docker, etc).
Both gentelella and flask-gentelella have a MIT license so they are completely free to use.
Thanks !


Error importing gitlab json schema

What version of CUE are you using (cue version)?

$ cue version
cue version v0.4.3 darwin/arm64

Does this issue reproduce with the latest release?
Yep
What did you do?
curl -JLO https://gitlab.com/gitlab-org/gitlab/-/raw/master/app/assets/javascripts/editor/schema/ci.json
cue import -f -l "#CISpec:" -p gitlab ci.json

Permalink to latest commit: https://gitlab.com/gitlab-org/gitlab/-/blob/7cac80c74d52cce38faa8a0068305a370a417ebd/app/assets/javascripts/editor/schema/ci.json
What did you expect to see?
The schema converted to CUE
What did you see instead?
constraint not allowed because type string is excluded:
    ./ci.json:116:33
constraint not allowed because type string is excluded:
    ./ci.json:703:25

Lines 116 and 703 for reference

@benmoss thanks for your patience here.
Following these steps to commit 7cac80c74d52cce38faa8a0068305a370a417ebd you reference above, using cad3f239 (current tip):
exec curl -sJLO https://gitlab.com/gitlab-org/gitlab/-/raw/7cac80c74d52cce38faa8a0068305a370a417ebd/app/assets/javascripts/editor/schema/ci.json
exec cue import jsonschema: ci.json

I see:
> exec cue import jsonschema: ci.json
[stderr]
constraint not allowed because type string is excluded:
    ./ci.json:68:33
constraint not allowed because type string is excluded:
    ./ci.json:577:29
[exit status 1]
FAIL: /tmp/testscript1311223080/repro.txtar/script.txtar:2: unexpected command failure

Looking at the relevant bits from the schema:
        "rules": {
          "type": "array",
          "items": {
            "anyOf": [
              {"type": "object"},
              {"type": "array", "minLength": 1, "items": { "type": "string" }}
            ],
            "properties": {
              "if": { "$ref": "#/definitions/if" },
              "changes": { "$ref": "#/definitions/changes" },
              "exists": { "$ref": "#/definitions/exists" },
              "variables": { "$ref": "#/definitions/variables" },
              "when": {
                "type": "string",
                "enum": ["always", "never"]
              }
            },
            "additionalProperties": false
          }
        }

The slightly cryptic error message refers to this line:
{"type": "array", "minLength": 1, "items": { "type": "string" }}

minLength is only something that applies to strings, not arrays. minItems is what is needed here, unless minLength was a constraint intended for the items in the array in which case:
{"type": "array", "items": { "type": "string",  "minLength": 1 }}

might have been what was intended.
If we fix the schema in these two places to use minItems it imports as expected.
So I believe this is a bug in the source JSONSchema. But please let me know if this analysis doesn't correspond with your understanding.

For anyone following along here, I just raised https://gitlab.com/gitlab-org/gitlab/-/merge_requests/148555 to fix the schema upstream.

And delighted to say that https://gitlab.com/gitlab-org/gitlab/-/merge_requests/148555 is now merged! cue import on the current tip of that schema now works!


spice-vdagent: 90s timeout on shutdown/reboot

Describe the bug
On (not quite latest master), spice-vdagent causes now a 90s downtime when powering off or rebooting my notebook. I didn't do that before.
To Reproduce
Steps to reproduce the behavior:

Enable Spice VDAgent in the configuration.nix:

    services.spice-vdagentd.enable = true;

Rebuild with a current master
Switch to the new generation and power off/reboot.

Expected behavior
"Immediate" powering off without 90s timeout.
Screenshots
https://images.sjau.ch/img/e28390be.jpg
Additional context
Add any other context about the problem here.
Notify maintainers

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

system: "x86_64-linux"
host os: Linux 5.4.31, NixOS, 20.09.git.0af01aa8d1fM (Nightingale)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.4
channels(hyper): "nixos-19.09pre185402.5eac339829b"
channels(root): "nixos-20.09pre220429.9b0d2f3fd15"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

Seems it's a general problem with currentl spice-vdagent:
https://bugzilla.redhat.com/show_bug.cgi?id=1813667
https://gitlab.freedesktop.org/spice/linux/vd_agent/issues/18 and according merge https://gitlab.freedesktop.org/spice/linux/vd_agent/-/merge_requests/7


Use the node name as default for the wallet name

What about to use the node name as default for the wallet name instead of the host name?
I think this make a lot of sense for who have an onion address in the host name.

Hi @bota87,
thanks for your suggestion.
I have thought about this as well. The problem is the node name is not part of the lnd connect string and therefore not known in that phase.
The connection has to be successful before we can query the node name.
Now if we add a new wallet for example and there is some connection problem, it would have no name at all.

Hi @michaelWuensch,
I was sure there is a technical explanation behind this choice.
Could you think of using the url as a temporary name until the first connection could be a solution?
However this request is not that important so feel free to just close it


Spanish translation

welcome section

The translation looks great :) thank you @azgcloudev
@rishabkumar7 looks like there is a conflict with some config stuff in _navbar and index.html. Can you resolve it and then merge changes please :)


Use SQLite as default test database

This PR switches the default testing database from MySQL to SQLite now that it is officially supported with #6.
The reason for this is so that developers and other contributors don't need to download and setup a MySQL environment to be able to run test suites.
Using SQLite as the default database engine for tests also improves the speed tests execute by ~1s (MySQL took 1.43 seconds vs 0.35s on SQLite using an M1 MacBook Pro)

seems need to match new workfllows run test for your new default config. can you fix that, or let me do it?

@l3aro on it, thanks!

@l3aro All tests have been resolved


Fixes pcalls, and cleans up tests with a macro.

The main thing here is that the usage of pcall was incorrect. pcall takes a variable number of functions, not forms, and returns a lazy list, which must be evaluated to guarantee that all the calls are complete.

@canweriotnow Would you mind reviewing this one?

@kytrinyx I will look at it tonight or tomorrow, my first reaction is that a macro is almost always the wrong thing, but I have to look at how these pcalls are working... my second reaction is that "almost" means the rest of the time, a macro is the only thing :smile:

Actually, (and especially since it's in the test), I like the solution, and the fact is we don't exposed exercists to macros much at all, so it's good exposure but I don't think it affects the understandability of the test (at least not to my Clojurian eyes).
I would like it if @alexkalderimis would add comments or otherwise document what the macro is doing for the uninitiated, but I don't see that as blocking the merge.


Release/0.2.0

Codecov Report

Merging #9 (afde4e2) into main (a0ea3fa) will increase coverage by 1.10%.
The diff coverage is 98.48%.

@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
+ Coverage   98.10%   99.21%   +1.10%
==========================================
  Files           9       11       +2
  Lines         264      382     +118
==========================================
+ Hits          259      379     +120
+ Misses          5        3       -2

Flag
Coverage Δ

unittests
99.21% <98.48%> (+1.10%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

tests/bom/test_pypi.py
100.00% <ø> (ø)

mdbom/mdbom.py
96.29% <87.50%> (+5.59%)
:arrow_up:

mdbom/bom/npm.py
100.00% <100.00%> (ø)

mdbom/bom/processor.py
97.67% <100.00%> (+0.79%)
:arrow_up:

tests/bom/test_npm.py
100.00% <100.00%> (ø)

tests/bom/test_processor.py
100.00% <100.00%> (ø)

tests/test_mdbom.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0ea3fa...afde4e2. Read the comment docs.


Minor issues in documentation & migration

Describe the bug
Not sure if this is right repo to reports issues in docs but here goes
I migrated from vgscollect:1.2.4 to vgscollect:1.6.6 and hit a build error with missing app:fieldType="cvc" on our VGSEditText. Seems it is gone in favor of new component CardVerificationCodeEditText. This took me a while to figure out since there was still in the VGSEditText docs mention it supports app:fieldType.
Also the main doc page showcasing the components says that CardVerificationCodeEditText is "Provides a user interface element that represents a person's name."
To Reproduce
Steps to reproduce the behavior:

Check links to documentation

Expected behavior
Would be also nice to have in the release notes a clear mention if a property is remove / replaced in a new version
Thanks

We updated our docs. Also, we mentioned those changes in the migration guide and release notes.

Thanks a lot!


feat: add get_observation and get_observation_for_place in client

This new feature fetches current observation from the v2/observation API endpoint.

hello @oncleben31 @dx44 @Quentame, could you please review ?


Roadmap: Role & member enforcement

Currently roles & user membership of tasks, teams, and projects don't actually do anything.
Will implement:

Organization admins can see everything
Organization members can only see projects, teams, and tasks they are admins or members of
Team admins can see all projects under team with read-write permissions
Team members can see all projects under team but only read-only permissions (will be configurable to allow team members read-write by default instead)
Project admins can manage project & add / manage users
Project members can manage tasks & task groups

Closed through #28


How do you get around self signed certs while updating?

Running cd "$(brew --repo)" && git fetch && git reset --hard origin/master && brew update
Results in
fatal: unable to access 'https://github.com/Linuxbrew/brew/': SSL certificate problem: self signed certificate in certificate chain

I can't see any flags or issues regarding this so I'm not sure what a work around is

Try
brew install curl

I'm not sure what your answer had to do with the issue. The problem is that I was trying to update brew while behind a corporate proxy/firewall (using websense). The cert is coming from websense intercepting traffic.
brew install curl git by default does not work
The curl problem is also documented in #275
You "solved" this by removing https for erlang but that doesn't work for libraries that aren't available through http (such as curl which is behind a https url)
Instead you must do echo insecure >> ~/.curlrc so that the -k option is not required.
brew install git works with the same trick as well.
brew update however does not work. I don't know why it still fails due the self signed cert. I haven't dug into what backend linux component/library is making the call that fails.

The problem is that I was trying to update brew while behind a corporate proxy/firewall (using websense). The cert is coming from websense intercepting traffic.

I was missing that key piece of information. I understand the problem now.

You "solved" this by removing https for erlang

That fix was specifically for erlang, whose SSL cert seems to be misconfigured, since it includes a self-signed cert in its chain, which curl rejects by default. I opened an issue upstream with openssl. They think it's a curl bug. https://github.com/openssl/openssl/issues/2833

Instead you must do echo insecure >> ~/.curlrc so that the -k option is not required.

Thanks for this work around! That's very helpful.

brew update however does not work. I don't know why it still fails due the self signed cert. I haven't dug into what backend linux component/library is making the call that fails.

You'll have to configure git to use insecure https URLs, as you did for curl.

Thanks @sjackman! I can confirm that first doing git config --global http.sslVerify false does indeed allow me to brew update now.

Thanks again for the workaround. That's very helpful to know.

To disable SSL verification
echo insecure >> ~/.curlrc
git config --global http.sslVerify false


Toyota/Lexus - Enhanced Speed Control - ACC speed setting

Hello, Thanks for a great version of openpilot with wonderful features!
I was curious if there were plans to expand the Enhanced Speed Control to Toyota/Lexus models.
I can help test if needed.
Car Make: Lexus
Car Model: NX300
Car Year: 2018
Hardware: Comma Three
Thanks,
Mark

Enhanced Speed Control works on all Toyota/Lexus platforms that use openpilot longitudinal control. It seems that your car by default uses stock Toyota longitudinal control. According to CARS.md, You will need to install the smartDSU to enable openpilot longitudinal control on your car.

Thanks for the reply. I had found the smartDSU hardware as being a requirement for actual longitudinal control however, I was under the impression that the enhanced speed control feature sent messages to the cars stock ACC to adjust the speed setting letting the stock system still do longitudinal control. That is what I was asking about getting working.

Thanks for the reply. I had found the smartDSU hardware as being a requirement for actual longitudinal control however, I was under the impression that the enhanced speed control feature sent messages to the cars stock ACC to adjust the speed setting letting the stock system still do longitudinal control. That is what I was asking about getting working.

Enhanced Speed Control requires openpilot longitudinal control or Custom Stock Longitudinal Control. Toyota/Lexus currently is not compatible with Custom Stock Longitudinal Control. For this platform, the best option is to install smartDSU and to allow openpilot longitudinal control to use Enhanced Speed Control.


Vim Editor Keybinding

Feature description:
Added vim mode. Click toggle vim button to switch. Each editor pane is handled independently.
Switch between normal/insert mode by ctrl+c. Not sure if I get it right but this post
https://github.com/brijeshb42/monaco-vim/issues/27#issue-470498046 seems to suggest it is
not trivial to get esc key to work as how vim users are used to.
UI changes:
Added toggle vim button(#vim-flag) and vim status bar(#v-status) on the editor pane.

Yep
On Mon, Aug 26, 2019 at 3:41 PM Austin Morton<EMAIL_ADDRESS>wrote:

I take it this PR is meant to supersede #1496
https://github.com/mattgodbolt/compiler-explorer/pull/1496?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mattgodbolt/compiler-explorer/pull/1555?email_source=notifications&email_token=AES2ND6R7XAVLNDADTXJCADQGRLY3A5CNFSM4IPWB4U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5F4XKA#issuecomment-525061032,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AES2NDZVWHCTMY7RDJSZGKTQGRLY3ANCNFSM4IPWB4UQ
.

Thanks for reviewing!
My apologies for some stuff that's not properly cleaned up.
Also no worries about not using these code in the end -- the work was more
on learning and it was enjoyable work anyway.


CB-15232. Support fluentd auto restart based on conditions.

details:

add new cdp_logging_agent_check.sh script to fluent salt definitions
script is uploaded on the fly in case of diagnostics + executed as well with a distribute parameter which distributes the script all across salt minion nodes
add new crontabs as well, it will run the script with doctor parameter daily and with distribute parameter on every week at Sunday
distribute command only distribute the script on nodes (where you do not have) + creates the crontabs on the minion nodes
with diagnostics you can fix existing clusters, for new clusters, salt highstate should create the crontabs + script

See detailed description in the commit message.

can be tested with these one liner on any salt-master node:
create thread dumps/file descriptor check etc.
curl -L https://raw.githubusercontent.com/hortonworks/cloudbreak/f220ace65879f5d6cb8e4116810049b9e8e98bf7/orchestrator-salt/src/main/resources/salt-common/salt/fluent/scripts/cdp_logging_agent_check.sh | bash -s -- dump

(it will create a large log file that can be used for troubleshooting)
distribute files /etc/crontab.d
curl -L https://raw.githubusercontent.com/hortonworks/cloudbreak/f220ace65879f5d6cb8e4116810049b9e8e98bf7/orchestrator-salt/src/main/resources/salt-common/salt/fluent/scripts/cdp_logging_agent_check.sh | bash -s -- distribute