WIP Python API

Resolves #538. Still a bit to be done:

[x] Accessing the RDD contents from Python still needs a bit of work, the core GenomicRDD APIs work though
[ ] I haven't added the region join or pipe APIs yet, because they do funny business with ClassTags/Implicits. The classTag bit will be easy to work around, just needs to be done.
[x] ./scripts/jenkins-test is almost definitely going to need an update
[x] Right now, this is on top of #1386, which has since merged. This PR needs a rebase.

Figured it'd be good to get some early comments.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1782/
Test PASSed.

It'll take me a bit to get to this.  This is also a pretty large patch...would there be a logical way to subdivide it at all?

Also, we don't have too much Python code from the past, but I'd be a pretty strong proponent of making PEP8 our official style, with possible exceptions if we want.

It'll take me a bit to get to this. This is also a pretty large patch...would there be a logical way to subdivide it at all?

Let me rebase; I think that will drastically drop the number of files touched.

Also, we don't have too much Python code from the past, but I'd be a pretty strong proponent of making PEP8 our official style, with possible exceptions if we want.

+1. I believe this is all PEP8, but haven't linted it.

Cool, check this one @0asa ;)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1786/
Test PASSed.

Slimmed this down a bit. Watch for another PR coming momentarily...

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1787/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 0b0f85d989ceb8f8fd48bafaf7d834a4d9dc4806 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 0b0f85d989ceb8f8fd48bafaf7d834a4d9dc4806 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0b0f85d989ceb8f8fd48bafaf7d834a4d9dc4806First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

This is failing in the Python part of the build as we don't have virtualenv installed on @AmplabJenkins. I will be following up with @shaneknapp.

we DO have conda environments available...  i'll be in the office tomorrow
and most of friday.
On Wed, Feb 15, 2017 at 4:22 PM, Frank Austin Nothaft <
<EMAIL_ADDRESS>wrote:

This is failing in the Python part of the build as we don't have
virtualenv installed on @AmplabJenkins https://github.com/AmplabJenkins.
I will be following up with @shaneknapp https://github.com/shaneknapp.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bigdatagenomics/adam/pull/1387#issuecomment-280187684,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABiDrMj3uCWuSy7R8LufF40wTjto1pRgks5rc5bjgaJpZM4L9omT
.

Thanks for the quick reply @shaneknapp! I had been assuming a virtualenv but it should be a trivial change to use conda instead. Let me give it a looksee tonight.

anaconda is in the default path (/home/anaconda/bin), so a simple 'which
conda' check would probably be sufficient.  i'd set specific versions of
packages, as the base installation is pretty old and stale.
On Wed, Feb 15, 2017 at 5:20 PM, Frank Austin Nothaft <
<EMAIL_ADDRESS>wrote:

Thanks for the quick reply @shaneknapp https://github.com/shaneknapp! I
had been assuming a virtualenv but it should be a trivial change to use
conda instead. Let me give it a looksee tonight.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bigdatagenomics/adam/pull/1387#issuecomment-280198812,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABiDrIUvcin0X_UpqHJ2NYSscFLQ-2Zeks5rc6RYgaJpZM4L9omT
.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1789/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 9e34eb6015c1fd36addadeb5fc0f1fec3d6a5b15 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 9e34eb6015c1fd36addadeb5fc0f1fec3d6a5b15 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 9e34eb6015c1fd36addadeb5fc0f1fec3d6a5b15First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1790/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 76b3ee7a54ba77a257fd5cd600a5a1c95958e518 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 76b3ee7a54ba77a257fd5cd600a5a1c95958e518 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 76b3ee7a54ba77a257fd5cd600a5a1c95958e518First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

here's some quick bash to fix this failure:
test -d /home/jenkins/.conda/envs/adam-build/ || conda create -n adam-build python=2.7 anaconda

Jenkins, test this please.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1791/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 624a670392588cb9de95103e1f3c16ff90a59d0b # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 624a670392588cb9de95103e1f3c16ff90a59d0b (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 624a670392588cb9de95103e1f3c16ff90a59d0bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

hmm, looks like there's a conda deactivate called before you call pip2.7.

after watching your build log scream by, i'd also recommend adding the "-q" flag to your conda create env call.  this will suppress all of the download status bars, etc.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1792/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains a31a5019257be7f4dabacd4fa0abd24d05d1fd4d # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision a31a5019257be7f4dabacd4fa0abd24d05d1fd4d (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f a31a5019257be7f4dabacd4fa0abd24d05d1fd4dFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1794/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 292c79d8eed09833ab30c7c558e67000f8501565 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 292c79d8eed09833ab30c7c558e67000f8501565 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 292c79d8eed09833ab30c7c558e67000f8501565First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1795/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains f331920d52d45616655d72a99a6ef21e82a3eca5 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision f331920d52d45616655d72a99a6ef21e82a3eca5 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f331920d52d45616655d72a99a6ef21e82a3eca5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1796/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 844b4108f8d2d5e308eff495092fb5c47750e768 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 844b4108f8d2d5e308eff495092fb5c47750e768 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 844b4108f8d2d5e308eff495092fb5c47750e768First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1797/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 864423412a7e73255fddc808719f4332ff9550e7 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 864423412a7e73255fddc808719f4332ff9550e7 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 864423412a7e73255fddc808719f4332ff9550e7First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1798/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 126e5d37160bd6f96888f73cf0fbbaf40f5d014e # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 126e5d37160bd6f96888f73cf0fbbaf40f5d014e (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 126e5d37160bd6f96888f73cf0fbbaf40f5d014eFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1799/Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1387/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 1e58cce84a5e7e6d85e00114ec6a1752c03483e8 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1387/merge^{commit} # timeout=10Checking out Revision 1e58cce84a5e7e6d85e00114ec6a1752c03483e8 (origin/pr/1387/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1e58cce84a5e7e6d85e00114ec6a1752c03483e8First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1800/Build result: FAILURE[...truncated 38 lines...]Triggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1801/Build result: FAILURE[...truncated 38 lines...]Triggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1802/Build result: FAILURE[...truncated 38 lines...]Triggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1803/
Test PASSed.

w00t! Thanks @shaneknapp for the help with the build bits.

Ping @laserson for review.

I want to experiment with this python API from the python or pyspark  REPL.
My first goal is to reproduce running the unit  tests manually at the REPL.
mvn built and test passes (as expected) fine for this branch.
I then started a virtualenv and ran
adam/adam-python$ python setup.py install

which seemed to work fine.
I then started pyspark repl from my spark-1.6.3 installation
~/Spark/v1/spark-1.6.3-bin-hadoop2.6/bin/pyspark

and import works fine in REPL
from bdgenomics.adam.adamContext import ADAMContex

then try to create the python ADAMContext instance
>>> ac = ADAMContext(sc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/bdgenomics/adam/adamContext.py", line 44, in __init__
TypeError: 'JavaPackage' object is not callable

Can you point me in the right direction?
Beyond the test code do we have any example ADAM python applications or REPL/notebook scripts?
I'd like to help make such once I can get it working here.

Hi @jpdna! A few things:

Did you mvn package? You'll need to recompile to get some of the Java-friendly method changes in this PR. (The initial ADAMContext required one of these changes)
Also, you need to set a few things on the PySpark command line/as an envar. Look in ./scripts/jenkins-test for details.

OK so I did mvn package and then reinstalled the adam-python module
and as for the envars inspired by ./scripts/jenkins-test
I tried so far
export PYTHONPATH=/home/paschalj/Spark/v1/spark-1.6.3-bin-hadoop2.6/python/lib/py4j-0.9-src.zip:${PYTHONPATH}

export PYSPARK_SUBMIT_ARGS="--jars /home/paschalj/frank_adam/python/adam/adam-assembly/target/adam_2.10-0.21.1-SNAPSHOT.jar --driver-class-path /home/paschalj/frank_adam/python/adam/adam-assembly/target/adam_2.10-0.21.1-SNAPSHOT.jar pyspark-shell"

But I am still getting the same TypeError: 'JavaPackage' object is not callable from ac = ADAMContext(sc) as described above in the pyspark REPL.
Do those two exports seem correct? And what else am I missing?
I am using a virtualenv and not anaconda.

It should work with a virtualenv, that's what I've been using locally. Do the tests pass for you when you make test in adam-python? (running mvn package -P python will run these)

Ah - yeah python build tests fail for me.
From a fresh clone/checkout of this branch issues/538-python I run:
mvn package -P python

and it fails.
I'll email you the console output with errors.
Before I had just ran mvn package earlier, not noticing if python module got compiled/test passed.
So, assuming a fresh git checkout - what steps needs to be done  to properly compile/test with the adam-python module?

I saw your email, will look more later. What version of spark are you running?

I saw your email, will look more later. What version of spark are you running?

Spark 1.6.3

I saw your email, will look more later. What version of spark are you running?

Spark 1.6.3

OK, that's what I'm running locally as well...

I tried from scratch again on different machine, with commands below, but got same errors during compile/test as I sent earlier.  Note, I tried running make develop in the adam-python dir first, but didn't seem to help.  Not sure when/if that needs to be run.
source venv/bin/activate
git clone https://github.com/fnothaft/adam.git
git checkout issues/538-python
export SPARK_HOME=/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6
export PYTHONPATH=/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/py4j-0.9-src.zip:{$PYTHONPATH}
pip install pytest
mvn package -P python

Error like below from mvn package -P python:
[INFO] ------------------------------------------------------------------------
[INFO] Building ADAM_2.10: Python APIs 0.21.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-versions) @ adam-python_2.10 ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-maven) @ adam-python_2.10 ---
[INFO]
[INFO] --- scalariform-maven-plugin:0.1.4:format (default-cli) @ adam-python_2.10 ---
[INFO] Modified 0 of 0 .scala files
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ adam-python_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/paschallj/adam_python/v2/adam/adam-python/src/main/resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ adam-python_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:exec (dev-python) @ adam-python_2.10 ---
pip install -e .
Obtaining file:///home/paschallj/adam_python/v2/adam/adam-python
Installing collected packages: bdgenomics.adam
  Found existing installation: bdgenomics.adam 0.21.1-SNAPSHOT
    Uninstalling bdgenomics.adam-0.21.1-SNAPSHOT:
      Successfully uninstalled bdgenomics.adam-0.21.1-SNAPSHOT
  Running setup.py develop for bdgenomics.adam
Successfully installed bdgenomics.adam
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ adam-python_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ adam-python_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/paschallj/adam_python/v2/adam/adam-python/src/test/resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ adam-python_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:exec (test-python) @ adam-python_2.10 ---
mkdir -p target
python2.7 -m pytest -vv --junitxml target/pytest-report.xml src
============================= test session starts ==============================
platform linux2 -- Python 2.7.12, pytest-3.0.6, py-1.4.32, pluggy-0.4.0 -- /home/paschallj/adam_python/v2/venv/bin/python2.7
cachedir: .cache
rootdir: /home/paschallj/adam_python/v2/adam/adam-python, inifile:
collecting ... collected 0 items / 5 errors

 generated xml file: /home/paschallj/adam_python/v2/adam/adam-python/target/pytest-report.xml
==================================== ERRORS ====================================
________ ERROR collecting src/bdgenomics/adam/test/adamContext_test.py _________
ImportError while importing test module '/home/paschallj/adam_python/v2/adam/adam-python/src/bdgenomics/adam/test/adamContext_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../venv/local/lib/python2.7/site-packages/_pytest/python.py:418: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../venv/local/lib/python2.7/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
src/bdgenomics/adam/test/__init__.py:26: in <module>
    from pyspark.context import SparkContext
E   ImportError: No module named pyspark.context
_____ ERROR collecting src/bdgenomics/adam/test/alignmentRecordRdd_test.py _____
ImportError while importing test module '/home/paschallj/adam_python/v2/adam/adam-python/src/bdgenomics/adam/test/alignmentRecordRdd_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../venv/local/lib/python2.7/site-packages/_pytest/python.py:418: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../venv/local/lib/python2.7/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
src/bdgenomics/adam/test/__init__.py:26: in <module>
    from pyspark.context import SparkContext
E   ImportError: No module named pyspark.context
_________ ERROR collecting src/bdgenomics/adam/test/featureRdd_test.py _________
ImportError while importing test module '/home/paschallj/adam_python/v2/adam/adam-python/src/bdgenomics/adam/test/featureRdd_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../venv/local/lib/python2.7/site-packages/_pytest/python.py:418: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../venv/local/lib/python2.7/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
src/bdgenomics/adam/test/__init__.py:26: in <module>
    from pyspark.context import SparkContext
E   ImportError: No module named pyspark.context
________ ERROR collecting src/bdgenomics/adam/test/genotypeRdd_test.py _________
ImportError while importing test module '/home/paschallj/adam_python/v2/adam/adam-python/src/bdgenomics/adam/test/genotypeRdd_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../venv/local/lib/python2.7/site-packages/_pytest/python.py:418: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../venv/local/lib/python2.7/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
src/bdgenomics/adam/test/__init__.py:26: in <module>
    from pyspark.context import SparkContext
E   ImportError: No module named pyspark.context
_________ ERROR collecting src/bdgenomics/adam/test/variantRdd_test.py _________
ImportError while importing test module '/home/paschallj/adam_python/v2/adam/adam-python/src/bdgenomics/adam/test/variantRdd_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../venv/local/lib/python2.7/site-packages/_pytest/python.py:418: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../venv/local/lib/python2.7/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
src/bdgenomics/adam/test/__init__.py:26: in <module>
    from pyspark.context import SparkContext
E   ImportError: No module named pyspark.context
!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 5 error in 0.28 seconds ============================
Makefile:82: recipe for target 'test' failed
make: *** [test] Error 2
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
	at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
	at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
	at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)
	at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)
	at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ADAM_2.10 .......................................... SUCCESS [  3.782 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [  5.201 s]
[INFO] ADAM_2.10: APIs for Java ........................... SUCCESS [  0.354 s]
[INFO] ADAM_2.10: CLI ..................................... SUCCESS [  1.197 s]
[INFO] ADAM_2.10: Assembly ................................ SUCCESS [  4.930 s]
[INFO] ADAM_2.10: Python APIs ............................. FAILURE [  1.578 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.204 s
[INFO] Finished at: 2017-02-27T08:16:10-05:00
[INFO] Final Memory: 224M/1313M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (test-python) on project adam-python_2.10: Command execution failed. Process exited with an error: 2 (Exit value: 2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :adam-python_2.10

@jpdna If you open a python interpreter and run from pyspark.context import SparkContext, what happens?

from pyspark.context import SparkContext

I get:
ImportError: No module named pyspark.context
I thought the way to use pyspark was via ./bin/pyspark for most part
If its needed how to I go about installing pyspark in my venv or adding it to python path?

@jpdna pyspark needs to be on your PYTHONPATH. Can you echo $PYTHONPATH and post here?

Earlier
PYTHONPATH=/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/py4j-0.9-src.zip

I tried adding pyspark so now:
echo $PYTHONPATH
/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/py4j-0.9-src.zip:/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip

so with that, now in normal python REPL from pyspark.context import SparkContext works.
However, in same venv, and same terminal sesion running
mvn package -P python -DskipTests

still fails with somewhat different verbose error messages that I will email to you.

still fails with somewhat different verbose error messages that I will email to you.

Can you post them here so I don't have to cross reference email threads? Thanks!

Sure - is there a way to force a code block veritical scroll bar?  I don't like having to make the page here so yuge
------------------------------------------------------------------------
[INFO] Building ADAM_2.10: Python APIs 0.21.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-versions) @ adam-python_2.10 ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-maven) @ adam-python_2.10 ---
[INFO]
[INFO] --- scalariform-maven-plugin:0.1.4:format (default-cli) @ adam-python_2.10 ---
[INFO] Modified 0 of 0 .scala files
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ adam-python_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/paschallj/adam_python/v2/adam/adam-python/src/main/resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ adam-python_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:exec (dev-python) @ adam-python_2.10 ---
pip install -e .
Obtaining file:///home/paschallj/adam_python/v2/adam/adam-python
Installing collected packages: bdgenomics.adam
  Found existing installation: bdgenomics.adam 0.21.1-SNAPSHOT
    Uninstalling bdgenomics.adam-0.21.1-SNAPSHOT:
      Successfully uninstalled bdgenomics.adam-0.21.1-SNAPSHOT
  Running setup.py develop for bdgenomics.adam
Successfully installed bdgenomics.adam
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ adam-python_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ adam-python_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/paschallj/adam_python/v2/adam/adam-python/src/test/resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ adam-python_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:exec (test-python) @ adam-python_2.10 ---
mkdir -p target
python2.7 -m pytest -vv --junitxml target/pytest-report.xml src
============================= test session starts ==============================
platform linux2 -- Python 2.7.12, pytest-3.0.6, py-1.4.32, pluggy-0.4.0 -- /home/paschallj/adam_python/v2/venv/bin/python2.7
cachedir: .cache
rootdir: /home/paschallj/adam_python/v2/adam/adam-python, inifile:
collecting ... collected 19 items

src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_alignments FAILED
src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_bed FAILED
src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_genotypes FAILED
src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_gtf FAILED
src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_interval_list FAILED
src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_narrowPeak FAILED
src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_sequence FAILED
src/bdgenomics/adam/test/adamContext_test.py::ADAMContextTest::test_load_variants FAILED
src/bdgenomics/adam/test/alignmentRecordRdd_test.py::AlignmentRecordRDDTest::test_save_as_bam FAILED
src/bdgenomics/adam/test/alignmentRecordRdd_test.py::AlignmentRecordRDDTest::test_save_sorted_sam FAILED
src/bdgenomics/adam/test/alignmentRecordRdd_test.py::AlignmentRecordRDDTest::test_save_unordered_sam FAILED
src/bdgenomics/adam/test/featureRdd_test.py::FeatureRDDTest::test_round_trip_bed FAILED
src/bdgenomics/adam/test/featureRdd_test.py::FeatureRDDTest::test_round_trip_gtf FAILED
src/bdgenomics/adam/test/featureRdd_test.py::FeatureRDDTest::test_round_trip_interval_list FAILED
src/bdgenomics/adam/test/featureRdd_test.py::FeatureRDDTest::test_round_trip_narrowPeak FAILED
src/bdgenomics/adam/test/genotypeRdd_test.py::GenotypeRDDTest::test_vcf_round_trip FAILED
src/bdgenomics/adam/test/genotypeRdd_test.py::GenotypeRDDTest::test_vcf_sort FAILED
src/bdgenomics/adam/test/genotypeRdd_test.py::GenotypeRDDTest::test_vcf_sort_lex FAILED
src/bdgenomics/adam/test/variantRdd_test.py::VariantRDDTest::test_vcf_round_trip FAILED

 generated xml file: /home/paschallj/adam_python/v2/adam/adam-python/target/pytest-report.xml
=================================== FAILURES ===================================
_____________________ ADAMContextTest.test_load_alignments _____________________

self = <bdgenomics.adam.test.adamContext_test.ADAMContextTest testMethod=test_load_alignments>

    def test_load_alignments(self):

        testFile = self.resourceFile("small.sam")
>       ac = ADAMContext(self.sc)

src/bdgenomics/adam/test/adamContext_test.py:30:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bdgenomics.adam.adamContext.ADAMContext object at 0x7fc4d4c57090>
sc = <pyspark.context.SparkContext object at 0x7fc4d4bf5790>

    def __init__(self, sc):
        """
            Initializes an ADAMContext using a SparkContext.

            :param pyspark.context.SparkContext sc: The currently active
            SparkContext.
            """

        self._sc = sc
        self._jvm = sc._jvm
>       c = self._jvm.org.bdgenomics.adam.rdd.ADAMContext(sc._jsc.sc())
E       TypeError: 'JavaPackage' object is not callable

src/bdgenomics/adam/adamContext.py:44: TypeError
...
...

src/bdgenomics/adam/adamContext.py:44: TypeError
----------------------------- Captured stderr call -----------------------------
17/02/27 13:21:12 INFO SparkContext: Running Spark version 1.6.3
17/02/27 13:21:12 INFO SecurityManager: Changing view acls to: paschallj
17/02/27 13:21:12 INFO SecurityManager: Changing modify acls to: paschallj
17/02/27 13:21:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(paschallj); users with modify permissions: Set(paschallj)
17/02/27 13:21:12 INFO Utils: Successfully started service 'sparkDriver' on port 44438.
17/02/27 13:21:12 INFO Slf4jLogger: Slf4jLogger started
17/02/27 13:21:12 INFO Remoting: Starting remoting
17/02/27 13:21:12 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:34489]
17/02/27 13:21:12 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 34489.
17/02/27 13:21:12 INFO SparkEnv: Registering MapOutputTracker
17/02/27 13:21:12 INFO SparkEnv: Registering BlockManagerMaster
17/02/27 13:21:12 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-ee0a1e40-7a0d-45a2-b0e9-4ae0e5628433
17/02/27 13:21:12 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
17/02/27 13:21:12 INFO SparkEnv: Registering OutputCommitCoordinator
17/02/27 13:21:12 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/02/27 13:21:12 INFO SparkUI: Started SparkUI at http://localhost:4040
17/02/27 13:21:12 INFO Executor: Starting executor ID driver on host localhost
17/02/27 13:21:12 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40674.
17/02/27 13:21:12 INFO NettyBlockTransferService: Server created on 40674
17/02/27 13:21:12 INFO BlockManagerMaster: Trying to register BlockManager
17/02/27 13:21:12 INFO BlockManagerMasterEndpoint: Registering block manager localhost:40674 with 511.1 MB RAM, BlockManagerId(driver, localhost, 40674)
17/02/27 13:21:12 INFO BlockManagerMaster: Registered BlockManager
17/02/27 13:21:12 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
17/02/27 13:21:12 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/02/27 13:21:12 INFO MemoryStore: MemoryStore cleared
17/02/27 13:21:12 INFO BlockManager: BlockManager stopped
17/02/27 13:21:12 INFO BlockManagerMaster: BlockManagerMaster stopped
17/02/27 13:21:12 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/02/27 13:21:12 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/02/27 13:21:12 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
17/02/27 13:21:12 INFO SparkContext: Successfully stopped SparkContext
17/02/27 13:21:12 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
========================== 19 failed in 17.87 seconds ==========================
Makefile:82: recipe for target 'test' failed
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
	at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
	at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
	at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)
	at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)
	at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ADAM_2.10 .......................................... SUCCESS [  4.471 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [  5.155 s]
[INFO] ADAM_2.10: APIs for Java ........................... SUCCESS [  0.378 s]
[INFO] ADAM_2.10: CLI ..................................... SUCCESS [  1.102 s]
[INFO] ADAM_2.10: Assembly ................................ SUCCESS [  4.967 s]
[INFO] ADAM_2.10: Python APIs ............................. FAILURE [ 19.141 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Thanks @jpdna! What does echo $PYSPARK_SUBMIT_ARGS show?

E.g., look at https://github.com/bigdatagenomics/adam/pull/1387/commits/d1d689c7be87080e4a4fc71bd320d47bc4289fd6#diff-af88c24e738248e4fdec0c4e4ab9d2ddR213.

success! - thanks @fnothaft
Now I can get on with trying out your  API!
it wasn't set before but now
echo $PYSPARK_SUBMIT_ARGS
--jars /home/paschallj/adam_python/v2/adam/adam-assembly/target/adam_2.10-0.21.1-SNAPSHOT.jar --driver-class-path /home/paschallj/adam_python/v2/adam/adam-assembly/target/adam_2.10-0.21.1-SNAPSHOT.jar pyspark-shell

echo $PYTHONPATH
/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/py4j-0.9-src.zip:/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip

launching via $SPARK_HOME/bin/pyspark still produced the java error I saw earlier,
but below worked fine in normal python REPL
from pyspark import SparkContext
from bdgenomics.adam.adamContext import ADAMContext
sc = SparkContext("local","jpappname")
ac = ADAMContext(sc)
reads = ac.loadAlignments("/home/paschallj/adam_python/v2/adam/adam-core/src/test/resources/small.sam")

I ran 'make develop' in adam/adam-pytthon first, but not sure if that was needed or not.

Ah, interesting! What's the command line you are using when you invoke $SPARK_HOME/bin/pyspark?

This PR should include a bump in coverage to the JavaADAMContext which has some ignored unit tests.

Any thoughts on getting this and #1391 into the 3/18 0.22.0 release? I think that it'd be desirable to get both in. Additionally, these two PRs have been open for a long time now, and we need to start getting review eyeballs on them at the least.

I'm attempting to use the `countKmers' function from the Python command line as below:
reads = ac.loadAlignments("small.sam")
myKmerCount = reads.countKmers(10)

But I can't seem to figure out how to then get out a map of Kmers and counts from the myKmerCount object.
This is one error I got from an attempt:
>>> myKmerCount.count()
2017-03-06 07:51:20 ERROR PythonRunner:95 - Python worker exited unexpectedly (crashed)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/worker.py", line 98, in main
    command = pickleSer._read_with_length(infile)
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 156, in _read_with_length
    length = read_int(stream)
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 545, in read_int
    raise EOFError
EOFError

	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
	at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
	at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)

Here Is another attempt:
 myKmerCount.collectAsMap()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/rdd.py", line 1520, in collectAsMap
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/rdd.py", line 772, in collect
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/rdd.py", line 142, in _load_from_socket
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 139, in load_stream
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 164, in _read_with_length
  File "/home/paschallj/Spark/1.6.3/spark-1.6.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 422, in loads
cPickle.UnpicklingError: invalid load key, 'A'.
>>> 2017-03-06 07:52:45 ERROR PythonRDD:95 - Error while sending iterator
org.apache.spark.SparkException: Unexpected element type class java.lang.Long
	at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:449)
	at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:452)
	at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:452)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)

I imagine this likely has to do with need to convert to some java/jvm object - but can you point me in right direction.  Also - in case users use dir(myKmerCount) to try to figure out what python functions will work, and find that some fail as above, we may need to describe in docs which python functions are available.

I'm going to close this in favor of #1391, which contains this same code and more.


[WIP] tensorflow: re-enable build from source

Motivation for this change
This resurrects the tensorflow source build. Its WIP because it builds an unreleased tensorflow version (current release didn't support the current bazel) and needs some cleanup, but it works. Builds upon https://github.com/NixOS/nixpkgs/pull/63580.
My motivation for this was that I need to run tensorflow on a VM without avx instructions while the official wheel requires avx (https://github.com/tensorflow/tensorflow/issues/17411). There are still cpus sold without AVX support, so we shouldn't require it by default.
Not sure yet if I'll actually push it over the finish line, since it "works for me" for now. But anyway, here it is.
CC @volth, @abbradar who have touched the source build before.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Of course a mix of perfectionism and sunk-cost fallacy drove me to continue to work on it.
I've included a tf 1.14 update and an update of the dependencies. Turns out the source build for 1.14 works after all, we just have to tell tensorflow not to check the maximum bazel version.
The value of taking some libs from the system is a bit questionable, but it's probably the right thing to do anyways.

Wow! That's really cool, I didn't expect anyone else having enough patience to dive into this :D
I'll hopefully be able to test during this week when replacement part for my PC arrives - Tensorflow is painfully slow to build otherwise.

Thanks! Apparently at least two people are https://github.com/NixOS/nixpkgs/pull/63208
This isn't quite done yet, had to debug a nasty issue with the cuda build (https://github.com/NixOS/nixpkgs/issues/38471#issuecomment-505189722). Then some cleanup.

Hm, can you explain why you need gcc and binutils in the bazel derivation for this to work? What’s the errors you are getting otherwise?

That is to fix the issue I explained here, where bazel tries to take the default locations for certain tools from PATH during build time.
Unfortunately that still doesn't fix the underlying problem, which is the build (only with cuda for some reason) failing because bazel tries to execute /usr/bin/ar. More details here: https://github.com/bazelbuild/bazel/issues/8715

Cuda build finally works! Cleanup still needed..

Some cleanup done.
@Profpatsch the bazel change ended up not being necessary here, since tensorflow sets its own toolchain. It may still be a good idea to fix up bazels default toolchain though. Unfortunately I couldn't figure out where the actual result ends up, so I couldn't verify my patches.

Forked out the update to https://github.com/NixOS/nixpkgs/pull/63873, because that is ready for merge now.

I work on merging both @timokau and @yorickvP's changes into one derivation which produces both a wheel and a library.

Thanks!
This PR isn't quite done however. The python2 build still sometimes fails, I'll probably need to add another fix to python2 (in addition to https://github.com/NixOS/nixpkgs/pull/64067). The reason I haven't worked on this is that I currently need the hardware for more important things (i.e. actually using tensorflow as opposed to packaging it). I'll likely get back to it some time, but if you want to take over I certainly won't complain either :)

Superseded by https://github.com/NixOS/nixpkgs/pull/64716, thanks @abbradar!


Translator Architecture Diagram Update

Given our conversation on yesterday's Architecture call, I updated the diagram reflecting our main themes.

The new version represents multiplicity for ARAs and KPs.
There's a new diagram to convey BTE's role.

Also capturing here that the group was ok with developing changes to an overall architecture diagram as a first step, deferring creation of detailed event traces to a future phase.
Comments welcome.

Looks good


Documentation error for transferPlayback

The documentation for spotifyApi.transferMyPlayback makes no comment of needing to pass a "play" option in but the code will error out without one.

@Simber1, you're absolutely right - the options parameter should be optional but currently isn't. I've fixed this locally and push a fix shortly. Thanks for raising it.

@Simber1, you're absolutely right - the options parameter should be optional but currently isn't. I've fixed this locally and push a fix shortly. Thanks for raising it.

bhdfs


Login attempt error after running locally for the first time [Invalid username or password]

My db is 10.6.5-MariaDB .
I confirmed that the manager data was normally created in the db.

The log-in information is as follows:
Username<EMAIL_ADDRESS>Password: password
Here are the results I tried:

Can I get information to log in?

I confirmed that the apiUrl option of shopizer-admin needs to be changed to localhost in order to run locally.
So close the issue


params: Node can be set for a private/other network type

ATM, when the node is initialised,  it is automatically trying to connect to bootstrap peers that are serving for devnet network.
We need more flexibility in setting up such parameter as swamp tests contains their own network and this is causing warnings on execution with messages like can not connect to bootstrap peers which are intended for another use-case.
This behaviour is also effecting our ephemeral-cluster.
Some refs can be found in params/networks.go

I thinking about ENV var that allows setting desired network preference

What was the reason to not make this part of the config instead?

Configs, according to my experience, are something that users often play around with a lot, trying different things, etc. The network switch in the config is something I expected users to touch and break subsequently breaking the synchronization. But I might be overcautious.

. The network switch in the config is something I expected users to touch and break subsequently breaking the synchronization.

I think proper documentation makes this a non-issue. Can also happen with env vars.


⚠ Jenkins Plugins Artifactory has degraded performance

In 4065804, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1187 ms

Resolved: Jenkins Plugins Artifactory performance has improved in d1d0210 after 12 minutes.


SSR tags get replaced on first render

Hello,
Thanx for this great library and the fix for FOUC.
Thast said, there is still a very small problem when using SSR + Client react
When using toComponent on the server render.
{head.title.toComponent()}
{head.meta.toComponent()}
{head.link.toComponent()}
{head.script.toComponent()}

The function will output a Element with the data-reactid attribute.
The when react is run on the client, these tags will then be replaced because the equation in updateTags failes return newElement.isEqualNode(existingTag);.
Because the new tag does not have adata-reactid attribute.
In the dropin replacement i created when there was that FOUC problem i could fix this, by removing the data-reactid attribute from the existing tags. (https://github.com/lemonCMS/react-helmet/blob/master/README.md). I hope this can help you.
Kind regards
Raymond

Thanks, good find.  Sounds like stripping out the data-reactid in updateTags, when that attribute exists, would be a good solution.

@lemonCMS are you planning on PRing this change? https://github.com/lemonCMS/react-helmet/commit/8efd8a55e89d2a7f0915dc563e3b09a2fe43988b

Heyo, so I tried rebasing @lemonCMS commit on top of HEAD and had some issues; it also didn't appear to fix the issue. (Also sorry for the reference spam)
I got this error when trying to build it against<EMAIL_ADDRESS>and babel@6:
warning.js:44 Warning: Element: `ref` is not a prop. Trying to access it will result in `undefined` being returned. If you need to access the same value within the child component, you should pass it as a different prop. (https://fb.me/react-special-props)

That may be unrelated though, as I'm depending on this inside of my protium package and npm-link is weird. Will continue to investigate, but would be awesome if someone who was already setup to build this could try it out (it's a very small change).


Release

@shobhitg I had to revert the recursion. I updated the extension (typescript hero) and the indexing process took forever and never stopped indexing. There are some errors in the code.


[Browser - Angular 10] Main Thread Blocked / UI Frozen due to aoa_to_sheet method

I have array of array with size of 0.5 Million rows, on converting into worksheet - Browser main thread is completely utilized for processing it and UI is blocked.
UI is Frozen for 5 to10 Seconds. Suggestion on Async way to perform this conversion will be awesome!
XLSXUtils.aoa_to_sheet(arr);

Async would not solve it on its own, but you can use sheetsJS in web worker

Some of the demos show how to use web workers.  They won't lock up the main thread.


Fix source link to "/inputnames" not "/inputtypes"

Fixed "input names" demo webpage URL for source link, since it currently points to the "input TYPES" source folder on GitHub.🤦♂

Thanks, and apologies for the late response!


test: Plumb the flaky flag from envoy_cc_test to the native.cc_test

This flag allows test to temporarily be marked flaky while test flakiness is being diagnosed and fixed, to reduce the likelihood of CI failing.
See https://docs.bazel.build/versions/2.0.0/be/common-definitions.html#common-attributes-tests for more info about the flaky flag.
Risk Level: Low
Testing: Unit Tests
Docs Changes: N/A
Release Notes: N/A
Signed-off-by: Yan Avlasov<EMAIL_ADDRESS>

Assigned @lizan in cause there's some reason we haven't done this earlier.

This seems like a good thing to have, though on the flip side it would probably mean we would never fix our flakes so I'm on the fence?

Do we want to use --runs_per_test_detects_flakes=true and --flaky_test_attempts as well? So it can give FLAKY status on CI.

@mattklein123  It maybe a better alternative to disabling the test, which is what happens when the reason for flakiness is not obvious or time consuming to fix. Disabled test could get stuck in limbo juts as the flaky attribute. The only difference is that if someone breaks the test outright it would fail even if marked flaky, but if it is disabled, nobody would know.
@lizan I think --runs_per_test_detects_flakes=true is only useful with the --runs_per_test=nnn flag which we do not use in CI.
I would argue against --flaky_test_attempts because it hides newly flaky tests by making CI pass. It could be an option to control CI cost if a lot of CI failures are due to newly flaky tests, so if there is a way to analyze this, this could be a good case for it.

Nm, my idea was using both of --runs_per_test_detects_flakes=true and --flaky_test_attempts so we can distinct FAILED and FLAKY, but seems FLAKY will make CI pass.

Let's merge this first, we can decide whether we want to use this later when we mark tests flaky. I'm trying to use this to make coverage less flaky.


feat: forgot password

The type of this PR is: feat
This PR resolves CX-1085
Description
Adds forgot password screen to the new login flow. Simplified slightly from spec to only allow submitting the form once without leaving and returning after discussion with @joanna20
https://user-images.githubusercontent.com/49686530/114462882-aeda7080-9bb1-11eb-8b65-7e818b9afb7a.mp4
PR Checklist (tick all before merging)

[x] I have included screenshots or videos to illustrate my changes, or I have not changed anything that impacts the UI.
[x] I have added tests for my changes, or my changes don't require testing, or I have included a link to a separate Jira ticket covering the tests.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added an app state migration, or my changes do not require one. (What are migrations?)
[x] I have added a CHANGELOG.yml entry or my changes do not require one.

this is great 🎉 thanks for adjusting the specs a bit for a better UX


travis tests failed in 4.16/main

as followup of #6113
3 tests failed in travis tests, see https://github.com/apache/cloudstack/pull/6113/checks?check_run_id=5536689365
26430.5	openjdk11	PATH=$HOME/.local/bin:$PATH	Linux	passed
 26430.6	openjdk11	PATH=$HOME/.local/bin:$PATH	Linux	errored
 26430.7	openjdk11	PATH=$HOME/.local/bin:$PATH	Linux	passed
 26430.8	openjdk11	PATH=$HOME/.local/bin:$PATH	Linux	errored
 26430.9	openjdk11	PATH=$HOME/.local/bin:$PATH	Linux	errored
 26430.10	openjdk11	PATH=$HOME/.local/bin:$PATH	Linux	passed
ISSUE TYPE

Bug Report

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

I see the following 2 tests failing on quite a few PRs  on main - I'll have a look at them -

component/test_project_limits: test_04_publicip_per_project | Status : FAILED ===
component/test_resource_limits:  test_01_publicip_per_domain | Status : FAILED ===


Add support for MLflow

A continuation of #5. Added documentation and tested locally.

Ah, that's curious. In any event, you can open a new issue in the spaCy repository.


Flutter push notifications for Desktop is required

Use Case : Desktop notifications like firebase notifications or flutter_local_notifications when flutter app is out of scope or closed .
Target platform :web _windows_macos_linux.
In windows can be done by getting URI from windows Notification service

In macOs ,there is notifications center

In linux

Feature required: adding implementations for adding notifications for desktop platforms

Hi @alionour,
When opening an issue, please try to provide all or some of the information indicated in the template.

Issue Template
If you are looking for support, please check out our documentation
or consider asking a question on Stack Overflow:
- https://flutter.dev/
- https://api.flutter.dev/
- https://stackoverflow.com/questions/tagged/flutter?sort=frequent

If you have found a bug or if our documentation doesn't have an answer
to what you're looking for, then fill our the template below. Please read
our guide to filing a bug first: https://flutter.dev/docs/resources/bug-reports

Steps to Reproduce

Please tell us exactly how to reproduce the problem you are running into.

Please attach a small application (ideally just one main.dart file) that
reproduces the problem. You could use https://gist.github.com/ for this.

If the problem is with your application's rendering, then please attach
a screenshot and explain what the problem is.

1. ...
2. ...
3. ...

Please tell us which target platform(s) the problem occurs (Android / iOS / Web / macOS / Linux / Windows)
Which target OS version, for Web, browser, is the test system running?
Does the problem occur on emulator/simulator as well as on physical devices?

Target Platform:
Target OS version/browser:
Devices:

Run your application with `flutter run --verbose` and attach all the
log output below between the lines with the backticks. If there is an
exception, please see if the error message includes enough information
to explain how to solve the issue.

Run `flutter analyze` and attach any output of that command below.
If there are any analysis errors, try resolving them before filing this issue.

Finally, paste the output of running `flutter doctor -v` here.

Closing this issue, since no information has been provided.
Thank you.

Description add

This would be something to implement as a third-party plugin, as it is on mobile. Especially since there are multiple possible servers someone might want to use for push notifications, and each would require a different plugin.

This would be something to implement as a third-party plugin, as it is on mobile. Especially since there are multiple possible servers someone might want to use for push notifications, and each would require a different plugin.

It is not about server notifications only it is about notifications created locally as result of schedule created on another thread not killed by app closure .
Flutter_local_notifications is not depending on servers yiu can create a push notification when sheduled time is reached

created on another thread not killed by app closure .

Threads don't outlive quitting an application; the feature you are describing uses OS services.

Flutter_local_notifications is not depending on servers

That's a third-party package as well; if you want it to support desktop you should file a feature request against that project.
Notifications are not built into the framework, or a first-party plugin, for mobile; desktop will be following the same model.

Yes ,I understood that thanks 😊 for your explanation


Undocumented behavior - Authorization Code requires Client Secret

Issue / Steps to reproduce the problem
When setting up a client for the Code-Flow without providing a client secret I get the error
Client secret is required for authorization_code, but no client secret is configured.
This behaviour is documented in an issue for IdentityServer3
but not in the current documentation for the token endpoint or the current documentation for the Authorization Code flow.
The behavior kind of contradicts the purpose of the PKCE-Flow for Authcodes as described by different sources as this is explicitly made for untrusted clients. We are trying to work around this , but I ask you to please
(a) Reconsider this decision when PKCE is enabled as libraries like this one do not send the secret on such a request and require workarounds.
(b) If you come to a negative conclusion please at least update the documentation to reflect this requirement.
I also filed an issue on the library I linked to provide the client_secret with this request as this probably also is not a perfect behavior - but also was not implemented on a previous issue.
Relevant parts of the log file
[16:11:13 VRB] Calling into client configuration validator: IdentityServer4.Validation.DefaultClientConfigurationValidator
[16:11:13 ERR] Invalid client configuration for client codeflow: Client secret is required for authorization_code, but no client secret is configured.

Thanks & best regards,
Christoph Sonntag

Why not set RequireClientSecret to false?
http://docs.identityserver.io/en/latest/reference/client.html

Because I did not read the documentation thoroughly enough 🙀. Thanks for your help :)


Bug: TypeError: cannot pickle 'staticmethod' object with a Field.default

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
For some reason, I can't use staticmethod with the default argument for a Field
Example Code
from collections.abc import Callable

from pydantic import BaseModel, Field

class SomeModel(BaseModel):
    @staticmethod
    def default_int_factory() -> int: ...

    # This is fine
    # int_factory: Callable[[], int] = default_int_factory
    # This blows up: TypeError: cannot pickle 'staticmethod' object
    int_factory: Callable[[], int] = Field(default=default_int_factory)

Python, Pydantic & OS Version
pydantic version: 2.9.2
        pydantic-core version: 2.23.4
          pydantic-core build: profile=release pgo=false
                 install path: /path/to/.venv/lib/python3.12/site-packages/pydantic
               python version: 3.12.5 (main, Aug 27 2024, 12:00:37) [Clang 15.0.0 (clang-15<IP_ADDRESS>)]
                     platform: macOS-14.5-arm64-arm-64bit
             related packages: fastapi-0.115.0 mypy-1.11.2 typing_extensions-4.12.2
                       commit: unknown

Interesting things are happening here. staticmethod is a descriptor. When it is referenced within the class definition (no matter if it wrapped in the Field call or not), the class doesn't exist yet and so:
class SomeModel(BaseModel):
    @staticmethod
    def default_int_factory() -> int: ...

    int_factory_1: Callable[[], int] = default_int_factory
    int_factory_2: Callable[[], int] = Field(default=default_int_factory)

int_factory_1 and int_factory_2.default are both instances of the staticmethod descriptor class. At some point in the BaseModel's metaclass' __new__ implementation, the class is created. Once this is done, we call collect_model_fields with the newly created class. In this function, we need to handle two cases:

annotated fields without any assignment, e.g. field: int.
annotated fields with an assignment, e.g. field: int = 2.

The second case is safely handled by this code:
https://github.com/pydantic/pydantic/blob/eda3eb472ce07e314858a7469e663f2e3c883578/pydantic/_internal/_fields.py#L196-L200
(not that default here has a poor naming as it refers to the assignment, it isn't necessarily a default value. For instance, you could have field: int = Field(description='test'), field is still required).
For both int_factory_1 and int_factory_2, we won't get any attribute error. However, for int_factory_1, because it is a staticmethod instance (and thus, a descriptor), doing getattr(cls, cls_name) results in the descriptor taking effect (and will thus return the underyling function, which can be deepcopied/pickled).

I think we either:

Align the Field(default=...) behavior with = ...., by calling the mimicking the class attribute access on the descriptor (that is, calling default.__get__(None, cls)). This avoids having to fix the FieldInfo.get_default function doing a deep copy of the default, which is the reason why it fails (this could be considered a Python bug, but according to this comment it seems to be intentional).
Align the = ... behavior with Field(default=...), that is changing default = getattr(cls, ann_name, PydanticUndefined) to default = cls.__dict__.get(ann_name, PydanticUndefined). This will require fixing the FieldInfo.get_default function.

To me, I think bullet 1 sounds better because it seems:

Bullet 1 more complies with the "principle of least surprise"
Bullet 2 will lead to different behaviors across Python versions, which is a headache
Bullet 2 involves breaking int_factory_1 from working

Related discussion for stdlib dataclasses: https://github.com/python/cpython/issues/102646


egui_glium: Flip vertical scroll direction on Linux

This is to address https://github.com/emilk/egui/issues/356
Draft until we have more data on exactly what is wrong where

In gnome wayland, this fix causes scrolling in the egui app to go in the opposite direction to all other apps. Without this fix, scrolling goes in the same direction as the entire desktop. Tested with natural scrolling enabled and disabled.

Let's leave it to winit to fix this then


Bug fix crash when encountering not supported agg function like last_value(bitmap)

Proposed changes
Issue Number: close #15256
The former logic inside aggregate_function_window.cpp would shutdown BE once encountering agg function with complex type like BITMAP. This pr makes it don't crash and would return one more concrete error message which tells the unsupported function signature to user.
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[x] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.04 seconds
load time: 632 seconds
storage size:<PHONE_NUMBER>5 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221221155337_clickbench_pr_66660.html

can disable the Bitmap input parameters on the FE,  in FunctionSet.java file

can disable the Bitmap input parameters on the FE, in FunctionSet.java file

Good idae, I'll change it in this way soon.  I highly suggest that we should keep the logic about checking if _fn_place_ptr is null in VAnalyticEvalNode::_destroy_agg_status, otherwise it might coredump once it's not initialized due to error exit inside VAnalyticEvalNode::prepare.


AKS/ACI - Add install-connector and remove-connector commands

Add az aks install-connector and az aks remove-connector commands
$ az aks install-connector --help

Command
    az aks install-connector: Deploy the ACI-Connector to a AKS cluster.

Arguments
    --name -n           [Required]: Resource name for the managed cluster.
    --name-aks          [Required]: The name of the AKS cluster. The name is case insensitive.
    --resource-group -g [Required]: Name of resource group. You can configure the default group
                                    using `az configure --defaults group=<name>`.
    --chart-url                   : URL to the chart, Default: https://github.com/Azure/aci-
                                    connector-k8s/raw/master/charts/aci-connector.tgz.
    --client-secret               : The secret associated with the service principal. If --service-
                                    principal is specified, then secret should also be specified. If
                                    --service-principal is not specified, the secret is auto-
                                    generated for you and stored in ${HOME}/.azure/ directory.
    --location -l                 : Location. You can configure the default location using `az
                                    configure --defaults location=<location>`.
    --os-type                     : Os type target, could be Windows, Linux or Both.  Default:
                                    Linux.
    --service-principal           : The service principal used for ACI authentication to Azure APIs.
                                    If not specified, it is created for you and stored in the
                                    ${HOME}/.azure directory.

Global Arguments
    --debug                       : Increase logging verbosity to show all debug logs.
    --help -h                     : Show this help message and exit.
    --output -o                   : Output format.  Allowed values: json, jsonc, table, tsv.
                                    Default: json.
    --query                       : JMESPath query string. See http://jmespath.org/ for more
                                    information and examples.
    --verbose                     : Increase logging verbosity. Use --debug for full debug logs.

$ az aks remove-connector --help

Command
    az aks remove-connector: Undeploy the ACI-Connector from an AKS cluster.

Arguments
    --name -n           [Required]: Resource name for the managed cluster.
    --name-aks          [Required]: The name of the AKS cluster. The name is case insensitive.
    --resource-group -g [Required]: Name of resource group. You can configure the default group
                                    using `az configure --defaults group=<name>`.
    --graceful                    : Mention if you want to drain/uncordon your aci-connector to move
                                    your applications running on ACI to your others nodes. Default :
                                    False.
    --os-type                     : Os type target, could be Windows, Linux or Both.  Default:
                                    Linux.

Global Arguments
    --debug                       : Increase logging verbosity to show all debug logs.
    --help -h                     : Show this help message and exit.
    --output -o                   : Output format.  Allowed values: json, jsonc, table, tsv.
                                    Default: json.
    --query                       : JMESPath query string. See http://jmespath.org/ for more
                                    information and examples.
    --verbose                     : Increase logging verbosity. Use --debug for full debug logs.

This checklist is used to make sure that common guidelines for a pull request are followed.
General Guidelines

[X] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

Command Guidelines

[X] Each command and parameter has a meaningful description.
[X] Each new command has a test.

(see Authoring Command Modules)

View a preview at https://prompt.ws/r/Azure/azure-cli/4996
This is an experimental preview for @microsoft.com users.
(It may take a minute or two for your instance to be ready)
Email feedback to 'azfeedback' with subject 'Prompt Feedback'.

I can't see where the command line options and docs are coming from in the Python, so I'll leave my notes here:

I don't understand the difference between the --name and --name-aks options. Can we draw out this distinction more clearly?
Same comment for --resource-group and --resource-group-name-aks
In uninstall-connector, --gracefull should be --graceful (one l)
In the --linux and --windows options, "Os" should be "OS" (capitalise acronyms)
Why is a target OS required for the 'remove' command?
...that makes me think about the OS switches in general - is it valid to pass both --linux and --windows?  If so, the explanation of the OS switches is confusing to me.  If not, we should look at changing the design, e.g. to a single --target-os or --os-type option.
--url-chart is documented as "The URL to the chart."  Which chart?
Subjective perhaps, but to me "URL chart" reads more like a chart of a URL or URLs.  The URL to a chart would be the "chart URL".  So perhaps consider changing this option to --chart-url?
In prose, a colon is normally written without a space before it, as in "Default: foo."  Is 'space before colon in default value docs' an Azure CLI convention?  If not, let's remove the space.

alright so --os for os types with the options windows, linux, both
then we are dropping the new rg for a connector and using the rg they created for the AKS cluster.
--url-chart is for the helm chart if they configured their own

Open question about defaults - for simplicity Linux, for coolness Windows + Linux.
Open to debate though.

@itowlson Thanks for the comments!

--name is the name of the connector and --name-aks is the name of the cluster where we want to deploy
I will automatically deploy the connector to the same RG where the AKS cluster is deployed, so I will remove the --resource-group-name-aks param.
the graceful typo should be fixed in the last commit
We need to specify the OS type for the remove command for the graceful command, since the we want to drain the nodes and the connector will have a different name depends if you are using windows or/and linux
I will change for ria --os windows --os linux and --os both
I will document the chart_url and applied your comments for the variable naming

@julienstroheker Thanks for the clarifications and for your thoughtful responses.  A couple of picky nits around the name and OS stuff, possibly due to my lack of understanding:

At the moment, the --name text says "Resource name for the managed cluster."  Will this change to say something like "Resource name for the ACI Connector instance." (or similar)?

I see what you're saying about the OS type in remove, but still don't have the mental model quite straight.  Are we saying the the name option is kind of a 'base name' that gets decorated with the OS name to make an actual name?  So if I pass --os both then I get two ACI Connectors, and I can remove them separately (e.g. can pass --os linux on remove to drain only the Linux connector)?  Pardon me if I'm being slow!

@itowlson

I am not sure to understand the question, are you talking about the description in the --help command ? if it the case, right now we have : The name for the ACI Connector but I can change it.
The way that it is coded right now in the connector, if you need windows (--os-type=Windows or --os-type=Both) you have no choice to use the docker image with the tag canary. That image will deploy 1 pod and 2 nodes. Those nodes will have the name aci-connector-0 and aci-connector-1. Since we cannot cleary identify which one is the windows one or the linux one we have no choice to delete both of them. This should be fixed soon, when we will have a better convention name we can apply your scenario.

Make senses ?

@julienstroheker That's cool, I didn't realise the --name text had been updated from the text posted at the top of the PR.  Your new text is fine.
Regarding why we need the OS type on remove, thanks for the offline conversation - I'll document it here so others have the information too.  The issue, as I understand it, is:

Deployment of the ACI Connector is controlled by a Helm chart.  az aks doesn't have low-level control over the nodes that get created.
Depending on whether the OSes include Windows, the Helm chart creates a single aci-connector node (Linux only) or two aci-connector-0/-1 nodes (Linux and Windows).  If Helm creates two nodes, it doesn't tell us which is which.
During removal, we could issue drain commands for all these nodes (aci-connector, aci-connector-0 and aci-connector-1) and handle "that node doesn't exist" errors.  But relying on node names feels brittle.
Therefore, we prefer the user to tell us what OSes they expect to be removing, and to infer from that what nodes to drain.
In future, it would be nice if the Helm chart labelled the nodes with their OS, or otherwise labelled or annotated them so we could locate them automatically.  Then we may be able to remove this requirement.

Let me know any errors or important omissions and I'll edit this comment so it can act as a reference.

@troydai  not sure who I need from the CLI team to get this merged

@devigned @JasonRShaver Can we get a review of this? thx!!

Is it possible to have --name be the name of the AKS managed cluster and --connector-name be the ACI connector's name instead? Then we use --name consistently across all the az aks commands, which seems less surprising to users.

@derekbekoe Thanks for the review ! I am on it !

@derekbekoe feel free to look at the new commits. Thanks

@derekbekoe Thanks again ! I applied your comments.


[Q] Is it possible to change PID number

I want to ask that is it possible to change the PID for video according to our need like the original PID is 0x0100 and i want to change that to 0x0101 or 0x0111?

Hi @jangrayogesh
The plugin remap does this. In your case, this means -P remap 0x0101=0x0111. The relevant PMT will be updated accordingly. Please RTFM for more details.

Thank You