[SPARK-7017][Build][Project Infra]: Refactor dev/run-tests into Python

All, this is a first attempt at refactoring dev/run-tests into Python. Initially I merely converted all Bash calls over to Python, then moved to a much more modular approach (more functions, moved the calls around, etc.). What is here is the initial culmination and should provide a great base to various downstream issues (e.g. SPARK-7016, modularize / parallelize testing, etc.). Would love comments / suggestions for this initial first step!
/cc @srowen @pwendell @nchammas

Test build #30950 has started for   PR 5694 at commit 3c53a1a.

Test build #30950 has finished for   PR 5694 at commit 3c53a1a.

This patch fails some tests.

This patch merges cleanly.

This patch adds no public classes.

This patch adds the following new dependencies:

tachyon-0.6.4.jar
tachyon-client-0.6.4.jar

This patch removes the following dependencies:

tachyon-0.5.0.jar
tachyon-client-0.5.0.jar

Looks like a great start! Left some comments mostly about Python style and organization. Will take a closer look next week at the actual logic and flow.

Just a naming suggestion - would be better to have an explicit .py file, and the shell file just calls that.

In the near future, I guess we want to move towards also converting the run-tests-jenkins script to Python so we can get rid of some nasty Bash-isms that have (necessarily) carried over into this port, like communicating via environment variables (e.g. the current testing block).

@rxin Thanks and taken care of!
@nchammas First, thanks a ton for all the Python reviews (I know it can be tedious)! Second, to your point about removing the Bash-isms, you're completely right in that I left them in for this PR such that we can get incremental improvement to the codebase. Once I tackle SPARK-7018 (e.g. dev/run-tests-jenkins) I think I'll be able to slowly move some of this old Bash necessity out. Feedback on that being the right path?

Yup, that sounds fine. We definitely don't have to tackle everything in one go, but eventually yeah we should phase out all the Bash-isms.
By the way, I don't see any recent updates from Jenkins on testing this PR since Jenkins is currently shutting down.
Just a heads up: Jenkins adds an additional layer of complexity in that it messes with the terminal and can cause weird problems with output buffering and whatnot.
So you'll definitely want to track how Jenkins is outputting results to screen as it runs tests to be sure things are working as expected.

Roger that. Let me look into using pipes.quote for the sbt output. Do we know what's up with Jenkins right now? I saw a thread a while back talking about a power outage at Berkeley, but thought I saw a message from Shane saying everything was back to normal. Is that not the case?

https://amplab.cs.berkeley.edu/jenkins/view/Spark/
Looks like planned maintenance?

Jenkins, test this please.

Oi Jenkins! Retest this please.

Test build #30974 has started for   PR 5694 at commit 07210a9.

Jenkinmensch, retest this please.

Test build #31062 has started for   PR 5694 at commit 26e18e8.

Test build #31062 has finished for   PR 5694 at commit 26e18e8.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.
This patch removes the following dependencies:

RoaringBitmap-0.4.5.jar
activation-1.1.jar
akka-actor_2.10-2.3.4-spark.jar
akka-remote_2.10-2.3.4-spark.jar
akka-slf4j_2.10-2.3.4-spark.jar
aopalliance-1.0.jar
arpack_combined_all-0.1.jar
avro-1.7.7.jar
breeze-macros_2.10-0.11.2.jar
breeze_2.10-0.11.2.jar
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-math-2.1.jar
commons-math3-3.4.1.jar
commons-net-2.2.jar
compress-lzf-1.0.0.jar
config-1.2.1.jar
core-1.1.2.jar
curator-client-2.4.0.jar
curator-framework-2.4.0.jar
curator-recipes-2.4.0.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
groovy-all-2.3.7.jar
guava-14.0.1.jar
guice-3.0.jar
hadoop-annotations-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-client-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-app-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-mapreduce-client-jobclient-2.2.0.jar
hadoop-mapreduce-client-shuffle-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-client-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
hadoop-yarn-server-common-2.2.0.jar
ivy-2.4.0.jar
jackson-annotations-2.4.0.jar
jackson-core-2.4.4.jar
jackson-core-asl-1.8.8.jar
jackson-databind-2.4.4.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-module-scala_2.10-2.4.4.jar
jackson-xc-1.8.8.jar
jansi-1.4.jar
javax.inject-1.jar
javax.servlet-3.0.0.v201112011016.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.10.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jline-2.10.4.jar
jodd-core-3.6.3.jar
json4s-ast_2.10-3.2.10.jar
json4s-core_2.10-3.2.10.jar
json4s-jackson_2.10-3.2.10.jar
jsr305-1.3.9.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.10.jar
kryo-2.21.jar
log4j-1.2.17.jar
lz4-1.2.0.jar
management-api-3.0.0-b012.jar
mesos-0.21.0-shaded-protobuf.jar
metrics-core-3.1.0.jar
metrics-graphite-3.1.0.jar
metrics-json-3.1.0.jar
metrics-jvm-3.1.0.jar
minlog-1.2.jar
netty-3.8.0.Final.jar
netty-all-4.0.23.Final.jar
objenesis-1.2.jar
opencsv-2.3.jar
oro-2.0.8.jar
paranamer-2.6.jar
parquet-column-1.6.0rc3.jar
parquet-common-1.6.0rc3.jar
parquet-encoding-1.6.0rc3.jar
parquet-format-2.2.0-rc1.jar
parquet-generator-1.6.0rc3.jar
parquet-hadoop-1.6.0rc3.jar
parquet-jackson-1.6.0rc3.jar
protobuf-java-2.4.1.jar
protobuf-java-2.5.0-spark.jar
py4j-<IP_ADDRESS>.jar
pyrolite-2.0.1.jar
quasiquotes_2.10-2.0.1.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.10.4.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar
scalap-2.10.4.jar
scalatest_2.10-2.2.1.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
snappy-java-<IP_ADDRESS>.jar
spark-bagel_2.10-1.4.0-SNAPSHOT.jar
spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
spark-core_2.10-1.4.0-SNAPSHOT.jar
spark-graphx_2.10-1.4.0-SNAPSHOT.jar
spark-launcher_2.10-1.4.0-SNAPSHOT.jar
spark-mllib_2.10-1.4.0-SNAPSHOT.jar
spark-network-common_2.10-1.4.0-SNAPSHOT.jar
spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
spark-repl_2.10-1.4.0-SNAPSHOT.jar
spark-sql_2.10-1.4.0-SNAPSHOT.jar
spark-streaming_2.10-1.4.0-SNAPSHOT.jar
spire-macros_2.10-0.7.4.jar
spire_2.10-0.7.4.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
tachyon-0.5.0.jar
tachyon-client-0.5.0.jar
uncommons-maths-1.2.2a.jar
unused-1.0.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.5.jar

jenkins, retest this please

Test build #31071 has started for   PR 5694 at commit 83e80ef.

Test build #31071 has finished for   PR 5694 at commit 83e80ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

@nchammas any other thoughts? I think we've got a pretty solid start wrt the refactor here.

I think as an initial refactoring, this is great. With this in place, we'll be able to refactor other lengthy Bash scripts involved in the build into Python, and add options that make it easy to run only specific groups of tests (e.g. SQL, or GraphX, etc.) as we discussed.
As far as this PR is concerned, I think one final thing to do is test some of the various failure modes that the script is supposed to cover:

Add a random syntax error in one of the files
Modify a test to always make it fail
Drop a license header from one of the files
etc.

For now you can do this manually by adding and then reverting commits to this PR, but this can in the future perhaps be captured in some kind of elementary script that tests the build itself.

jenkinbox, retest this please.

Merged build triggered.

Merged build started.

Test build #31162 has started for   PR 5694 at commit b0b2604.

Test build #31162 has finished for   PR 5694 at commit b0b2604.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31162/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #31169 has started for   PR 5694 at commit 803143a.

Test build #31169 has finished for   PR 5694 at commit 803143a.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31169/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #31190 has started for   PR 5694 at commit a5bd445.

Test build #31190 has finished for   PR 5694 at commit a5bd445.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31190/
Test FAILed.

Merged build finished. Test FAILed.

@brennonyork In the commit that says "removed license file for SparkContext", shouldn't we have seen a specific error about "RAT checks failed" as opposed to the generic "some tests failed"?
From what I remember when I added those specific error messages to the testing setup, the only time we should see "some tests failed" is when something failed in a weird way.
License checks or unit tests failing should always yield specific error messages.

@nchammas good catch, turns out there was a bug with what error code was being returned to properly have dev/run-tests-jenkins reporting the correct error message. Just pushed up a fix to hopefully handle this.

jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #31297 has started for   PR 5694 at commit 7613558.

Test build #31297 has finished for   PR 5694 at commit 7613558.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31297/
Test FAILed.

jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #31309 has started for   PR 5694 at commit b37328c.

Test build #31309 has finished for   PR 5694 at commit b37328c.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31309/
Test FAILed.

jenkins, retest this please

Merged build triggered.

Merged build started.

Test build #31319 has started for   PR 5694 at commit 56d3cb9.

Test build #31319 has finished for   PR 5694 at commit 56d3cb9.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31319/
Test FAILed.

jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #31326 has started for   PR 5694 at commit e4a96cc.

Test build #31326 has finished for   PR 5694 at commit e4a96cc.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.
This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31326/
Test FAILed.

This patch fails RAT tests.

There we go. :)

:+1: jenkins, lets keep this going, retest this please.

Merged build triggered.

Test build #31330 has started for   PR 5694 at commit 983f2a2.

Test build #31330 has finished for   PR 5694 at commit 983f2a2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31330/
Test FAILed.

Merged build finished. Test FAILed.

jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #31332 has started for   PR 5694 at commit f041d8a.

Test build #31332 has finished for   PR 5694 at commit f041d8a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31332/
Test FAILed.

jenkins, looking for a break in mima! retest this please!

Merged build triggered.

Merged build started.

Test build #31338 has started for   PR 5694 at commit d825aa4.

Test build #31338 has finished for   PR 5694 at commit d825aa4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31338/
Test FAILed.

@nchammas just wanted to get out a rolling count of what we've tested thus far:

Failed tests
Failed MiMa excludes
Failed scala style checks
Failed Apache RAT checks

Will continue today to hopefully finish up the last bit! Let me know if I missed anything!

Oh, we also showed a failed build scenario too. Going to finish with PySpark tests and SparkR tests.

jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #31428 has started for   PR 5694 at commit 9a592ec.

jenkins, for the last time, retest this please

Merged build triggered.

Merged build started.

Test build #31433 has started for   PR 5694 at commit afeb093.

Test build #31428 has finished for   PR 5694 at commit 9a592ec.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31428/
Test FAILed.

Test build #31433 has finished for   PR 5694 at commit afeb093.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31433/
Test FAILed.

jenkins, retest this please

Merged build triggered.

Merged build started.

Test build #31570 has started for   PR 5694 at commit b1ca593.

Test build #31570 has finished for   PR 5694 at commit b1ca593.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31570/
Test PASSed.

@pwendell @srowen @nchammas @rxin can I get a final review on this? It's LGTM and I've tested all the major error cases we need this script to report on. Recognize this is a highly critical piece of code when it comes to the whole Spark ecosystem though so I'd rather make sure we get more eyes on it / nits taken care of now before we look to merge into master.

@shaneknapp forgot to add you into this as well (sorry)! Esp. since you're dealing with stdout and stderr issues right now I want to make sure this doesn't add any excess bloat to that (shouldn't...).

Just an FYI: I can't speak on their behalf, but it's likely that the committers will want to hold off on merging this until after the 1.4 feature freeze (which I think is next week).

@nchammas -- totally understood!  :)
@brennonyork -- i'm looking forward to spending some time digging in to this code next week

Merged build triggered.

Merged build started.

Test build #32436 has started for   PR 5694 at commit 703f095.

Merged build finished. Test FAILed.

Test build #32436 has finished for   PR 5694 at commit 703f095.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32436/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #32440 has started for   PR 5694 at commit f950010.

Test build #32440 has finished for   PR 5694 at commit f950010.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32440/
Test PASSed.

Now that the branch-1.4 was cut, could I get a few eyes on this one? :)

yes absolutely, i will put some eyes on this!
a quick question:  is there a specific version of python we want to run this under?  currently it will be anaconda python 2.7.  your code is pretty version agnostic, and i'm just curious.

@shaneknapp honestly I hadn't thought about it, but since this should be capable of running on developers' boxes I would assume we should keep it agnostic. Is there a specific version of Python that Spark dictates is needed just for builds? If so we should match to that I would say.

Python 2.6 is the oldest version of Python that Spark officially supports.
We also added Python 3 support recently, so ideally this script should be able to run on 2.6+ and 3.3+, but I think it's fine to start with just 2.6+ since this is a developer script.

could I get a few eyes on this one?

Doesn't look like you've made any big changes since my last review, so I can say this patch LGTM overall.
You more recent commit mentions SPARK-6908; I'm guessing you pulled those changes from here.
You might also want to incorporate the changes proposed in #5955 by @yhuai.

@nchammas -- the tests run against 2.6.6, 2.7.9 (pypy 2.5.1) and 3.4.3.  for this script 2.6 or 2.7 will be fine, and will actually execute with 2.7.9 in our test environment:
[root@amp-jenkins-worker-03 ~]# /usr/bin/env python
Python 2.7.9 |Anaconda 2.1.0 (64-bit)| (default, Mar  9 2015, 16:20:48)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>>

@nchammas you're correct on SPARK-6908. I hadn't pulled in #5955 as it wasn't already closed / merged so wasn't sure if that was something the committers wanted or not. Assumed it was, but figured I'd wait to see. If there's consensus on that though I'll be happy to add it in.

@rxin @pwendell @srowen could I get a few more eyes on this? Getting tricky to keep fixing merge conflicts and backporting them into this script :)

cc @davies - who really knows Python.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33089/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #33093 has started for   PR 5694 at commit f9deba1.

Test build #33093 has finished for   PR 5694 at commit f9deba1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33093/
Test PASSed.

Merged build finished. Test PASSed.

This looks good to me, just one minor comment.

Merged build triggered.

Merged build started.

Test build #33262 has started for   PR 5694 at commit b1248dc.

Test build #33262 has finished for   PR 5694 at commit b1248dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33262/
Test PASSed.

Merged build finished. Test PASSed.

Thanks @davies, anyone have any comments / concerns?

outside of the merge conflicts, lgtm!
On Thu, May 21, 2015 at 2:08 PM, Brennon York<EMAIL_ADDRESS>wrote:

Thanks @davies https://github.com/davies, anyone have any comments /
concerns?
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/5694#issuecomment-104420452.

Merged build triggered.

Merged build started.

Test build #33546 has started for   PR 5694 at commit d90ab2d.

Test build #33546 has finished for   PR 5694 at commit d90ab2d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33546/
Test PASSed.

Bump on this thread.
/cc @pwendell

Can you update the Python lint checks to run on this file as well?  I noticed that it fails pep8 checks:
pep8 dev/run-tests.py
dev/run-tests.py:34:62: W291 trailing whitespace
dev/run-tests.py:41:74: W291 trailing whitespace
dev/run-tests.py:43:1: W293 blank line contains whitespace
dev/run-tests.py:45:53: W291 trailing whitespace
dev/run-tests.py:93:20: E203 whitespace before ':'
dev/run-tests.py:94:20: E203 whitespace before ':'
dev/run-tests.py:95:20: E203 whitespace before ':'
dev/run-tests.py:96:20: E203 whitespace before ':'
dev/run-tests.py:103:62: W291 trailing whitespace
dev/run-tests.py:104:22: W503 line break before binary operator
dev/run-tests.py:108:22: W503 line break before binary operator
dev/run-tests.py:151:65: W291 trailing whitespace
dev/run-tests.py:153:48: E261 at least two spaces before inline comment
dev/run-tests.py:154:57: E261 at least two spaces before inline comment
dev/run-tests.py:155:45: E261 at least two spaces before inline comment
dev/run-tests.py:157:44: W291 trailing whitespace
dev/run-tests.py:197:45: W291 trailing whitespace
dev/run-tests.py:198:50: W291 trailing whitespace
dev/run-tests.py:199:59: W291 trailing whitespace
dev/run-tests.py:268:24: W291 trailing whitespace
dev/run-tests.py:283:60: W291 trailing whitespace
dev/run-tests.py:288:33: W291 trailing whitespace
dev/run-tests.py:289:43: W291 trailing whitespace
dev/run-tests.py:296:57: W291 trailing whitespace
dev/run-tests.py:317:40: W291 trailing whitespace
dev/run-tests.py:320:79: W291 trailing whitespace
dev/run-tests.py:332:44: W291 trailing whitespace
dev/run-tests.py:342:44: W291 trailing whitespace
dev/run-tests.py:353:1: W293 blank line contains whitespace
dev/run-tests.py:392:79: W291 trailing whitespace
dev/run-tests.py:404:1: W293 blank line contains whitespace

Hey @brennonyork - sorry it took so long to review this. I can iterate on this much more closely over the next few days. I took a close look at this. This is a faithful porting of the bash implementation, but I think we should leverage Python to make things a bit nicer than they could be in bash. As part of re-writing this in python I think we should so some much needed simplification of the construction of all the various permutations of test configurations. I sketched an outline as to how it can work, let me know if you have any questions. Without those changes I think this will remain really hard for someone else to understand.

@pwendell thanks for the review! You're certainly correct in that I took a "just get it into Python and working first" approach. Was unsure whether we wanted more of what you had laid out above or something that got it into Python and then incrementally built upon, but glad that clarification is there now. Will move forward fixing the comments you and @JoshRosen supplied and get a new commit back soon!

Given that there's some messiness in reading bash error code scripts here, etc. do you think it would be a net reduction of complexity to just include run-tests-jenkins in this refactoring as well?

Hey Josh- I had this thought too, but I figured we can do it incrementally.
On Thu, Jun 4, 2015 at 10:32 PM, Josh Rosen<EMAIL_ADDRESS>wrote:

Given that there's some messiness in reading bash error code scripts here,
etc. do you think it would be a net reduction of complexity to just include
run-tests-jenkins in this refactoring as well?
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/5694#issuecomment-109167677.

Build triggered.

Build started.

Test build #34540 has started for   PR 5694 at commit 1f607b1.

Test build #34540 has finished for   PR 5694 at commit 1f607b1.

This patch fails some tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34548 has started for   PR 5694 at commit 1ecca26.

Test build #34548 has finished for   PR 5694 at commit 1ecca26.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34551 has started for   PR 5694 at commit db7ae6f.

Test build #34551 has finished for   PR 5694 at commit db7ae6f.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34599 has started for   PR 5694 at commit eb684b6.

pep8 checks still need to be enabled/fixed for this file:
[joshrosen Spark (1add396...)]$ pep8 dev/run-tests.py
dev/run-tests.py:34:62: W291 trailing whitespace
dev/run-tests.py:41:74: W291 trailing whitespace
dev/run-tests.py:43:1: W293 blank line contains whitespace
dev/run-tests.py:45:53: W291 trailing whitespace
dev/run-tests.py:93:20: E203 whitespace before ':'
dev/run-tests.py:94:20: E203 whitespace before ':'
dev/run-tests.py:95:20: E203 whitespace before ':'
dev/run-tests.py:96:20: E203 whitespace before ':'
dev/run-tests.py:103:62: W291 trailing whitespace
dev/run-tests.py:104:22: W503 line break before binary operator
dev/run-tests.py:108:22: W503 line break before binary operator
dev/run-tests.py:151:65: W291 trailing whitespace
dev/run-tests.py:153:48: E261 at least two spaces before inline comment
dev/run-tests.py:154:57: E261 at least two spaces before inline comment
dev/run-tests.py:155:45: E261 at least two spaces before inline comment
dev/run-tests.py:157:44: W291 trailing whitespace
dev/run-tests.py:197:45: W291 trailing whitespace
dev/run-tests.py:198:50: W291 trailing whitespace
dev/run-tests.py:199:59: W291 trailing whitespace
dev/run-tests.py:268:24: W291 trailing whitespace
dev/run-tests.py:283:60: W291 trailing whitespace
dev/run-tests.py:288:33: W291 trailing whitespace
dev/run-tests.py:289:43: W291 trailing whitespace
dev/run-tests.py:296:57: W291 trailing whitespace
dev/run-tests.py:317:40: W291 trailing whitespace
dev/run-tests.py:320:79: W291 trailing whitespace
dev/run-tests.py:332:44: W291 trailing whitespace
dev/run-tests.py:342:44: W291 trailing whitespace
dev/run-tests.py:353:1: W293 blank line contains whitespace
dev/run-tests.py:392:79: W291 trailing whitespace
dev/run-tests.py:404:1: W293 blank line contains whitespace

Whoops, my bad: had an outdated checkout.  Pep8 looks fine.

Merged build triggered.

Merged build started.

Merged build triggered.

Merged build started.

Test build #34603 has started for   PR 5694 at commit 60b3d51.

Test build #34603 has finished for   PR 5694 at commit 60b3d51.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build finished. Test FAILed.

jenkins retest this please

Merged build triggered.

Merged build started.

Test build #34605 has started for   PR 5694 at commit 60b3d51.

Test build #34605 has finished for   PR 5694 at commit 60b3d51.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #34599 has finished for   PR 5694 at commit eb684b6.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

@shaneknapp I'm pretty sure you're talking about the check here which will test for AMPLAB_JENKINS being set in the environment. Let me know if I'm wrong here!

jenkins retest this please

Merged build triggered.

Merged build started.

Test build #34612 has started for   PR 5694 at commit 60b3d51.

Test build #34612 has finished for   PR 5694 at commit 60b3d51.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

@brennonyork -- yep, you're right.  :)

Any idea what's up with this Python style check failure?
========================================================================
Running Python style checks
========================================================================
Python lint checks failed.
*** Error compiling './examples/src/main/python/kmeans.py'...
  File "./examples/src/main/python/kmeans.py", line 71
    lambda (p1, c1), (p2, c2): (p1 + p2, c1 + c2))
           ^
SyntaxError: invalid syntax

[error] running ['./dev/lint-python'] ; received return code 1

Could dev/lint-python be using a different Python version in a way that's breaking things?  That looks like a Python-3-like error, since I think Python 3 dropped support for tuple unpacking in lambdas, so are we maybe using Python 3 to run lint-python as of this patch?

Alternatively, we could just update the example to work in Python 3, which we should probably do anyways to ensure that Python 3 users have a good out-of-the-box experience.

Yeah you nailed it. I was about to push up a bug fix that should fix that, but I like your idea better of just updating the example. Turns out moving the os.environ["PATH"] set to include python3 in the PATH before all python checks was failing it.

Merged build triggered.

Merged build started.

Test build #34617 has started for   PR 5694 at commit 705d12e.

Test build #34617 has finished for   PR 5694 at commit 705d12e.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34618 has started for   PR 5694 at commit 03fdd7b.

btw, we are currently setting the PATH variable for python3/anaconda in the both this script as well as in the build config on jenkins.  this makes for a sad shane...  i'd really like us to only do this once, and if we do it in the build config it'll take some (albeit minor) complexity out of run-tests.py.  thoughts?

I would say thats a great idea since its already set that Spark supports Python3. As long as devs know that all python scripts will run under python3 by default it would simplify this (and likely other bash->python scripts coming).

Test build #34618 has finished for   PR 5694 at commit 03fdd7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34630 has started for   PR 5694 at commit b7c72b9.

Test build #34630 has finished for   PR 5694 at commit b7c72b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

@pwendell @JoshRosen thoughts on the initial refactor? I've incorporated an, albeit minimal, additional set of test checks for MLlib, GraphX, and Streaming.

After thinking about it a bit more, I think that this PR's test triggering logic could be significantly easier to understand if we rewrote it in terms of a job / dependency graph abstraction.
At a high level, we have Spark modules / components which

are affected / impacted by file changes (e.g. a module is associated with a set of source files, so changes to those files change the module),
contain a set of tests to run, which are triggered via Maven, SBT, or via Python / R scripts.
depend on other modules and have dependent modules: if we change core, then every downstream test should be run.

Right now, the per-module logic is spread across a few different places: we have one function that describes how to detect changes for all modules, another function that (implicitly) deals with module dependencies, etc.
Instead, I propose that we introduce a class for describing a module, use instances of this class to build up a dependency graph, then phrase the "find which tests to run" operations in terms of that graph.  I think that this will be easier to understand / maintain.

Test build #908 has started for   PR 5694 at commit b7c72b9.

Test build #908 timed out     for PR 5694 at commit b7c72b9     after a configured wait of 175m.

After some reflection, I think that the suggestions outlined in my last high-level comment might take a while to implement since they're essentially suggesting a different high-level organization of some of the test-selection logic. Although I think in the long-term we may want a design closer to what I proposed there, for now I think we should set that aside and just focus on making the smaller changes that I suggested to get the current design working. If we do that, I'll commit this and will submit a followup PR to explore my reorganization ideas.
I'll go take a look at my previous comments now and will flag anything that doesn't need to be done as part of this PR.

Maybe we can skip this for now, but if I could add one additional test module / component I would add one for docs/ so that documentation-only changes don't cause a full test.

Merged build triggered.

Merged build started.

Test build #34874 has started for   PR 5694 at commit ec1ae78.

Test build #34874 has finished for   PR 5694 at commit ec1ae78.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34955 has started for   PR 5694 at commit aa03d9e.

Test build #34955 has finished for   PR 5694 at commit aa03d9e.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.


Automate planned tests page

At the moment, whenever there is a planned test we need to update and deploy gov.uk/alerts manually.
We also update the site manually to announce upcoming public tests, like Reading.
This commit:

automatically puts a banner on the homepage and updates the ‘Planned tests’ page whenever there is an operator test happening on a given day
gives us a way of publishing details of upcoming public tests without having to directly edit HTML (it’s yet another YAML file)

I’ve even made it work in the case when there’s an upcoming public test, and a current operator test, like we had in the run-up to Reading.
For operator tests the banner will stay up until the end of the day. I think this is sensible since the test might not last for very long, and there might be several in one day (which we treat as one). For this to work in a completely automated way we will need to schedule republishing of the site at 00:01 each day.
If we do this then the MNOs don’t need to arrange their planned tests with us in advance, reducing our support burden.
With some fake data:

Sorry for the mega commit

@CrystalPea when operators are doing tests we are not going to announce these in advance. They will just appear as a ‘planned test’ at the same time as they happen, then stay on the site for the rest of the day.
We’ll only use the YAML file for announcing tests on public channels in advance.
We probably need some stuff in the runbook for this.


Postrgest app logging

Having a play with Postgrest it seems really cool but I haven't seen any logging happening to stdout.
Is it possible to add in logging for what port/ip the server is running on, and optionally (based on log level perhaps) log the requests and responses received by the server?
It would be good to have this log to stdout/stderr by default so it will work well with docker...

Absolutely, good idea. I'll add a log level argument and that supports error, info, and debug levels. What would the ideal format be for log messages in your opinion?

How about something like this: http://en.wikipedia.org/wiki/Common_Log_Format ?
Better to go with someone that is common to make it easier for log analyzers, eg Splunk.

This looks great.
I totally agree that we should add logging, although I am going to prioritize this lower than some other open issues.

It turned out to be really easy to do so I just added some basic logging.
It could still be improved though because it does not show the user identifier when someone makes an authenticated request.


Add Deneyap 5x7 LED Matrix Arduino Library Manager

@ArduinoBot

Hello @zeynepdicle. The automated system here only operates on open pull requests. Since your submission was already accepted and the pull request merged, the bot won't pay any more attention to this pull request.
But this human is happy to provide assistance. Was there something you needed from the bot?

I want to upload this Arduino library new release 1.0.1

After the initial addition of the library, the indexer job periodically checks each of the libraries in the index and picks up any new release/tag that meets the requirements listed here:
https://github.com/arduino/library-registry/blob/main/FAQ.md#what-are-the-requirements-for-publishing-new-releases-of-libraries-already-in-the-library-manager-list
This means there is no need to take any additional action to add new releases of your library to Library Manager. They will be added automagically! It does take a little time for the indexer job to run, and then some additional time after that for the update to propagate through the CDN and the local cache. But it will get there.
You can monitor the indexing of the library releases by checking the library's dedicated indexer logs web page:
https://github.com/arduino/library-registry/blob/main/FAQ.md#user-content-can-i-check-on-library-releases-being-added-to-library-manager


Documentation is down

Link to documentation is dead.

I mirrored the docs: http://sod.all-zippy.com/
Just until Per is back ;)

🎉


Scanned location geolocation, fallback iplocation

Hello
New feature;
If someone scanns the QR Code, it should ask for permission to use his gps location with geolocation.
if he says no, use iplocation.
i tried my self to implemend it, but failed
Greetings
remo

Hello could be a useful feature. Unfortunately, due to commitments related to my studies and my work, I am unable to add new features. I invite you to try again and once I have tested your code I will add the new functionality to the master branch


fix: broken live button on hlx3 without prod host

Fix #65
You can test the fix with a sidekick from: https://sidekick-issue-65--helix-pages--adobe.hlx.page/tools/sidekick/?giturl=https://github.com/adobe/business-website

:tada: This PR is included in version 3.7.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Way to disable large folders warning

This issue is about request to allow a configurable way to disable notification

This is reported previously in https://github.com/microsoft/vscode/issues/62871, https://github.com/Microsoft/vscode/issues/31188 but locked and cannot continue discussion in original issue.
Clarify, this is not due to misconfiguring excludes:
I'm working on one of repo currently have about 6500 js + jsx + ts + tsx files and it'll likely to grow over. Even exclude unnecessary files as much, vs code keep displaying same notification and would like to disable it, or either way to increase maximum files tsserver can handle to avoid.

@kwonoj Can you please share more information about your general project structure. As a starting point, with your workspace open in VS Code, run code --status from the command line and share the output.
That notification means that intellisense will not work properly, so we'd like to understand why it shows up in the first place instead of just hiding it

I wonder what the upper limit for number of files is?
I'm currently at 2492 .ts files.
|    Folder (some-project): 12865 files
|      File types: ts(5060) map(5026) js(2559) json(123) tsbuildinfo(48)
|                  md(30) jpg(3) env(2) sql(2) gitignore(2)
|      Conf files: tsconfig.json(52) package.json(1) tslint.json(1)
|                  launch.json(1) settings.json(1)
|      Launch Configs: node(2)

@mjbvz
Version:          Code 1.37.0 (036a6b1d3ac84e5ca96a17a44e63a87971f8fcc8, 2019-08-08T01:22:37.660Z)
OS Version:       Darwin x64 18.6.0
CPUs:             Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz (8 x 2500)
Memory (System):  16.00GB (1.87GB free)
Load (avg):       2, 3, 3
VM:               0%
Screen Reader:    no
Process Argv:     .
GPU Status:       2d_canvas:                     enabled
                  flash_3d:                      enabled
                  flash_stage3d:                 enabled
                  flash_stage3d_baseline:        enabled
                  gpu_compositing:               enabled
                  multiple_raster_threads:       enabled_on
                  native_gpu_memory_buffers:     enabled
                  oop_rasterization:             disabled_off
                  protected_video_decode:        unavailable_off
                  rasterization:                 enabled
                  skia_deferred_display_list:    disabled_off
                  skia_renderer:                 disabled_off
                  surface_synchronization:       enabled_on
                  video_decode:                  enabled
                  viz_display_compositor:        disabled_off
                  webgl:                         enabled
                  webgl2:                        enabled

CPU %	Mem MB	   PID	Process
    0	    82	 66730	code main
    0	    66	 66763	   gpu-process
    0	   246	 66764	   window (js/modern/infra/console.js — webapp)
    0	   131	 66770	     extensionHost
    0	    98	 66772	       electron_node tsserver.js
    0	   229	 66773	       electron_node tsserver.js
    0	    66	 66783	         electron_node typingsInstaller.js typesMap.js
    0	    82	 66776	       electron_node eslintServer.js
    0	    49	 66832	       /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /Applications/Visual Studio Code.app/Contents/Resources/app/extensions/json-language-features/server/dist/jsonServerMain --node-ipc --clientProcessId=66770
    0	    66	 66771	     watcherService
    0	    49	 66784	     searchService
    0	   131	 66780	   shared-process
    0	     0	 67048	     /bin/ps -ax -o pid=,ppid=,pcpu=,pmem=,command=

Workspace Stats:
|  Window (js/modern/infra/console.js — webapp)
|    Folder (webapp): more than 40616 files
|      File types: png(17140) php(11060) json(4695) js(2470) api(1106)
|                  md(1030) hbs(818) less(424) jpg(292) bcmap(168)
|      Conf files: makefile(6) package.json(6) tsconfig.json(4)
|                  webpack.config.js(3) launch.json(1) settings.json(1)
|                  gulp.js(1)
|      Launch Configs: chrome

Open to provide tsserver trace or some more detail, but cannot attach in here as this is private repo.

Have you tried to set "disableSizeLimit": true inside compilerOptions in tsconfig.json (or in jsconfig.json if you use it)?

@IllusionMH omg I just learned those option and it seems it makes notification go away, thx for tips.
Should I close issue? I think so as it's controllable at least.

@mjbvz maybe it make sense to update https://code.visualstudio.com/Docs/languages/javascript "Common questions" section to describe steps how to deal with this error?
How to exclude large folders and then disableSizeLimit after common large folders are excluded?
For some reason I can't find where docs are strored and how to contribute

That sounds like a good idea. Here's the relevant repo: https://github.com/microsoft/vscode-docs/blob/master/docs/typescript/typescript-compiling.md
Would you like to submit a docs PR that adds that section?

Yes, will do if no one will do it before me.

Glad you found the setting!
I will issue my standard warning that that message is in place because generally TS Server will crash due to running out of memory when analyzing that much code at once, so simply turning off the warning and proceeding will likely result in a loop of TS Server trying to analyze everything, crashing, and restarting. But if it works for you anyway - carry on!

I will issue my standard warning that that message is in place because generally TS Server will crash due to running out of memory when analyzing that much code at once, so simply turning off the warning and proceeding will likely result in a loop of TS Server trying to analyze everything, crashing, and restarting.

@RyanCavanaugh
Thanks for heads up - this is actually living concern for us. But also working on large-scale codebase we don't have lot of options as well. What'd be recommended way to scale up tsserver to deal with those large amount of files (without warning)? in a long run we'd like to split project into project references, but that won't going to happen soon.


Value Mappings are not working properly with v0.71

Hi,
I just updated the panel from version 0.7 to 0.71. In version 0.7 I am using the panel for showing the status of my house windows. Therefore I needed the type String and  a value mapping from true -> open and false -> closed. These values (closed, open)  I used for the thresholds. After the update the panel does not accept the mapped values, it only works when I am using the  origin values (true, false) from the database. So I think the value mapping does not work properly.
Thanks & regards,
Michael

Fixed in 0.8.0

0.8.0 published


Infinite loop when devtools are open

Describe the bug
When the devtools are open while the application is being started, the page forcibly reloads constantly.
Link to the blitz that caused the error
https://stackblitz.com/edit/shopify-hydrogen-zrvke1?file=src%2Fcomponents%2FInstantSearch%2FSearch.client.jsx&title=Hydrogen
Steps to reproduce

Open in a new tab
Open devtools
reload

Expected behavior
nothing special, you can debug
Parity with Local

[X] I have run the project in my local machine and I could not reproduce the issue.

Screenshots
https://user-images.githubusercontent.com/6270048/167124467-1a948011-0865-4bb5-b18b-91af77a7cd9e.mov
Platform
Browser name  = Chrome
Full version  = 103.0.5043.0
Major version = 103
navigator.appName = Netscape
navigator.userAgent = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5043.0 Safari/537.36
performance.memory = {
  "totalJSHeapSize": 110918319,
  "usedJSHeapSize": 107567123,
  "jsHeapSizeLimit":<PHONE_NUMBER>
}
Hash = f2b430413f5697306be55fc5c653053509a24007

Additional context
No response

Hey @Haroenv 👋 . Thanks for reporting! So the problem here might be that you had the Update on reload checkbox checked on the Application > Service workers tab? This causes our service worker to be uninstalled and reinstalled again and again. I'll create a ticket internally because someone else reported this a while back as well. We have to find out why this is causing a loop.

I just checked and I can indeed still reproduce on that URL, but Application > Service workers doesn't have Update on reload checked as far as I can tell. I tried it with Google Chrome Canary, Version 113.0.5622.0 (Official Build) canary (x86_64)

That's pretty strange. I'm not able to reproduce this. I'll check with some team members to see if someone else also runs into this.


Defender Network Protection Instructions

For the setup instructions for Open Network Detection - options 1 and 2 are both listed as Enable. The 3 options are Enable, Disable and Audit. Which one is Audit, 1 or 2?

DefenderOpenNetworkDetection
0
1 - enable, 0 - disable, 2 - Enable. This setting is managed by an IT Admin to enable, audit, or disable open network detection. In Audit mode, alerts will be sent only to the ATP portal with no user side experience. For user experience, set the config to "Enable" mode.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 6d4eae37-e4cf-2642-d2cf-c648f90d2cc6
Version Independent ID: 7cb88605-ddc9-5a1a-67f4-a28da1819227
Content: Configure Microsoft Defender for Endpoint on iOS features
Content Source: microsoft-365/security/defender-endpoint/ios-configure-features.md
Service: microsoft-365-security
Sub-service: mde
GitHub Login: @mjcaparas
Microsoft Alias: macapara

@derekb519 Thank you for your feedback. We have made changes to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/12650
Once the author approves, the changes will be live.
Thanks
Sri


Frontend / Backend Increment

Backend:

Added a common code base that can be used
Added jwt validation / generation into the usermgmt service
Make a root gradle project for the Backend
Add a first FeignClient for synchronous communication between services
Fix Authorization Stripping in API Gateway
Add user-db
Functional signup Endpoint
Functional signin Endpoint

Frontend:

Send Booking
Signin
Signup
History Screen

One important note, that we discovered during development:
It seems like the gradlew bootJar jar command does not always cause a rebuilt of the jars. We had to delete the build folders to actually have new jars generated and get the docker containers working correctly.


Explain how to set TypeScript up for using with jk

I have been using jk with TypeScript exclusively since I adopted it in January (for both compiling node.js to be consumed by jk directly and publishing compatible node modules to npm) and would be happy to contribute something back to the project.  What kind of explanation were you thinking, a walk through under examples/ or an explicit section of the docs?

That's a generous offer, thank you! "How To" documentation -- a guide, walkthrough, worked example -- would be the most useful, I reckon. I would put it in the docs, under the heading "How to" in the navigation.
The way the docs get made into a website is a little tricky to trace through. Basically: content goes in https://github.com/jkcfg/website/tree/master/static/docs, and navigation is defined in https://github.com/jkcfg/website/blob/master/src/pages/navigation.json. You can run the website locally -- instructions in the README https://github.com/jkcfg/website/blob/master/README.md.


Add support for having each header have its own color

This could be implemented by adding GetTitleColor or similar  to ISGTabbedPagerDatasource.
This would allow to have different colors for each selected header.

I might do just that, we'll see just how badly our client wants it 😃


Some figures disappear when restarting the server

I have a job generating figures in a certain env of my visdom server. When i restart the server and refresh the browser visdom page, some figures just disappear.
Everything looks fine before and after restarting the server in the logs. Any clue for why this is happening ? Someone already had similar issue ?

I have checked the .json files, it seems like the env where i have this issue i can't find data related to the disappearing figures (even before restarting the server).

Hi @erakra - the server doesn't save environments by default, you'll have to use the save button as described in the docs. As for the mathmenu/mathzoom issue, this is a known problem that just hasn't been solved yet, but it has no impact on the functionality of visdom.

Thanks @JackUrb ! My environment was being saved actually. The error might have been coming from several jobs interacting with several environments. I fixed it by resetting these and well aligning them to different environments.


Mesh comparison: ReplicaViewer to check mesh

Hallo,
thanks for your great job!
I want to compare different reconstructed mesh at same view for Replica Dataset. But actually, at Replica official website they provide the mesh without texture. They provide also ReplicaViewer to check rendered mesh. However, it is hard to adjust recontructed mesh in meshlab with ReplicaViewe.
Is it possible to get textured mesh from  ReplicaViewer? How can you check them like Figure 3. in paper?
I really appreciate any help you can provide!
Dakai

Hello,
Many thanks for your interest in our work. This was actually a problem that we were struggling with during our experiments, and we did not find a solution for it.
In the end, for qualitative comparison, we captured the rendered scene from ReplicaViewer from camera parameters that approximately was visually consistent with the meshes in MeshLab. In other words, we manually tuned the image-capturing position in ReplicaViewer with the MeshLab.
Kind regards,


[BUG] Longhorn does not launch on systems with a non-standard path

Describe the bug
Linux distributions such as NixOS use a very non-standard path (e.g /run/wrappers/bin:/root/.nix-profile/bin:/etc/profiles/per-user/root/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bi) - in NixOS's case, this is to allow for use of the Nix package manager.
On these systems, because Longhorn uses nsenter to enter the host namespace, and nsenter uses the path of the namespaces that called it, basic system utilities such as mount, as well as iscsiadm can't be found or called (produces Failed to execute: nsenter [–mount=/host/proc/*/ns/mnt mount], output nsenter: failed to execute mount: No such file or directory and Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host during initial startup of the longhorn-manager pod).
I have built some extremely scrappy derived images based on the current stable release of Longhorn at https://github.com/duckfullstop/nixos-longhorn - these simply append the necessary NixOS-specific path entries to the container's path, such that nsenter can find the requisite binaries properly. This is not a complete fix - there is, for some reason, a hardcoded lookup path in longhorn-manager/csi/nfs/nsenter.go that means the share-manager pod cannot properly instantiate ReadWriteMany Persistent Volumes. However, even without patching that, the containers with the updated path work perfectly, and Longhorn works flawlessly (excepting the ReadWriteMany issue).
The affected Longhorn components are longhorn-manager and longhorn-instance-manager, as these are the only two images that make nsenter calls.
I'm not 100% sure what the best and cleanest way to resolve this is, so I'm opening this as an issue instead of a load of PRs.
To Reproduce

Bootstrap a NixOS Kubernetes cluster (see https://nixos.wiki/wiki/Kubernetes for some vague instruction, I have incantations that I'm using that I can post somewhere eventually but a basic setup will experience the same issue)
Ensure that openiscsi is available in the host environment with config.environment.systemPackages = [ pkgs.openiscsi ];
Deploy the current Longhorn manifest to the cluster
Be sad when longhorn-manager fails to come up

Expected behavior
longhorn-manager proceeds with initialisation.
Log
During startup, if not using containers with patched $PATH (log from longhorn-manager):
Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host
Failed to execute: nsenter [–mount=/host/proc/*/ns/mnt mount], output nsenter: failed to execute mount: No such file or directory

When attempting to use a ReadWriteMany Persistent Volume Claim with patched $PATH (log from longhorn-csi-plugin/longhorn-csi-plugin):
level=error msg="GRPC error: rpc error: code = Internal desc = Failed to create nsenter executor, err: unable to find required binary mount on host"

Environment:

Longhorn version: v1.1.0 / v1_20201216
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: NixOS v1.20.1 (custom derivation)

Number of management node in the cluster: 1
Number of worker node in the cluster: 2

Node config

OS type and version: NixOS 20.09
CPU per node: 8
Memory per node: 16G
Disk type(e.g. SSD/NVMe): SSD
Network bandwidth between the nodes: 1GB/s

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal

Additional context
The cleanest way that I can personally think of to resolve this one is to add a variable to longhorn-manager and longhorn-instance-manager that allows for appending to the system path, and to update csi/nfs/nsenter.go to check against this path instead of the hardcoded choices.

Feel free to submit a PR for it. It will be great if Longhorn can work with NixOS
For the nsenter path modification, you can make this hard code path configurable, then you can simply modify the YAML files to make Longhorn fit NixOS. And besides the line (csi/nfs) you mentioned in the post, I think you may also need to modify repo longhorn/go-iscsi-helper, which is a dependency of other Longhorn repos.
Here is the brief/possible solution:

Modify the related nsenter functions in longhorn/go-iscsi-helper. Then push the commit to your private repo.
Update file go.mod in repo longhorn/longhorn-manager and repo longhorn/longhorn-engine. Then run go mod tidy && go mod vendor in the root directory of the repos to update the dependency.
Make the nsenter path configurable for:

longhorn-manager,
longhorn-csi-plugin dynamically deployed by longhorn-driver-deployer
longhorn-instance-manager deployed by longhorn-manager

Make sure longhorn-csi-plugin (csi/nfs), longhorn-engine (it's a kind of process running in longhorn-instance-manager pods), and longhorn-manager will use the modified nsenter path.
Run make to build new images for longhorn-manager and longhorn-engine
Update the YAML files in directory longhorn/longhorn-manager/deploy.

Feel free to submit a PR for it. It will be great if Longhorn can work with NixOS
For the nsenter path modification, you can make this hard code path configurable, then you can simply modify the YAML files to make Longhorn fit NixOS. And besides the line (csi/nfs) you mentioned in the post, I think you may also need to modify repo longhorn/go-iscsi-helper, which is a dependency of other Longhorn repos.
Here is the brief/possible solution:

Modify the related nsenter functions in longhorn/go-iscsi-helper. Then push the commit to your private repo.
Update file go.mod in repo longhorn/longhorn-manager and repo longhorn/longhorn-engine. Then run go mod tidy && go mod vendor in the root directory of the repos to update the dependency.
Make the nsenter path configurable for:

longhorn-manager,
longhorn-csi-plugin dynamically deployed by longhorn-driver-deployer
longhorn-instance-manager deployed by longhorn-manager

Make sure longhorn-csi-plugin (csi/nfs), longhorn-engine (it's a kind of process running in longhorn-instance-manager pods), and longhorn-manager will use the modified nsenter path.
Run make to build new images for longhorn-manager and longhorn-engine
Update the YAML files in directory longhorn/longhorn-manager/deploy.

I'll happily take a look into putting some PRs together! Should have some time to delve it at the end of this week, otherwise hopefully next week.

I'll happily take a look into putting some PRs together! Should have some time to delve it at the end of this week, otherwise hopefully next week.

Took 5 to look into how I'm going to implement this: changing the container's $PATH is probably the least painful but least clean, otherwise I could change the codepath of go-iscsi-helper's Execute() function to make an additional set of calls to find the binaries in a given set of extra paths (basically a wrapper around os/exec's LookPath(), which queries against $PATH), and then do nsenter [args] /i/found/this/path/and/it/contains/mount.
I'll keep thinking about it, not sure which way to go because $PATH is arguably the authority on system path, having an additional wrapper over the top of it feels rather janky. If I do the former, I could also just add an EXTRA_PATH envvar to the containers, and then have the container's bootstrap merge that into the system path (a bit like what my containers are doing right now) - would still need to patch https://github.com/longhorn/longhorn-manager/blob/master/csi/nfs/nsenter.go#L80 as previously mentioned though.

Took 5 to look into how I'm going to implement this: changing the container's $PATH is probably the least painful but least clean, otherwise I could change the codepath of go-iscsi-helper's Execute() function to make an additional set of calls to find the binaries in a given set of extra paths (basically a wrapper around os/exec's LookPath(), which queries against $PATH), and then do nsenter [args] /i/found/this/path/and/it/contains/mount.
I'll keep thinking about it, not sure which way to go because $PATH is arguably the authority on system path, having an additional wrapper over the top of it feels rather janky. If I do the former, I could also just add an EXTRA_PATH envvar to the containers, and then have the container's bootstrap merge that into the system path (a bit like what my containers are doing right now) - would still need to patch https://github.com/longhorn/longhorn-manager/blob/master/csi/nfs/nsenter.go#L80 as previously mentioned though.

Working on this now, should have some preliminary PR's out by today or tomorrow.
I note the existence of @joshimoo's https://github.com/longhorn/longhorn-manager/commit/b467f4fbb9f936b91830449a125cea04c9b0b6b6 which seems to be working in a similar direction to the one I am, except way more cleanly; perhaps they could chime in with their intentions?

Working on this now, should have some preliminary PR's out by today or tomorrow.
I note the existence of @joshimoo's https://github.com/longhorn/longhorn-manager/commit/b467f4fbb9f936b91830449a125cea04c9b0b6b6 which seems to be working in a similar direction to the one I am, except way more cleanly; perhaps they could chime in with their intentions?

Having dived a bit deeper into this, I think implementing this one is going to be a bit beyond my pay grade, at least without sticking random calls to os.getEnv() everywhere (which was making me throw up a bit). I started trying to add a new ExtraPath attribute to the spec of longhorn.InstanceManager, but there was so much cascading change that had to be done to do that (to longhorn.Node, etc) that I'm not really confident submitting change of that scale to something I'm not intimately familiar with.
If you can provide some further guidance I'd be happy to assist, but for the time being I'd like to sit back and see what @joshimoo was working on with their commit.
I do have a preliminary changeset for longhorn/longhorn-instance-manager that tweaks the nsenter stuff to use the system path instead of hardcoded paths - happy to push that if it would assist.

Having dived a bit deeper into this, I think implementing this one is going to be a bit beyond my pay grade, at least without sticking random calls to os.getEnv() everywhere (which was making me throw up a bit). I started trying to add a new ExtraPath attribute to the spec of longhorn.InstanceManager, but there was so much cascading change that had to be done to do that (to longhorn.Node, etc) that I'm not really confident submitting change of that scale to something I'm not intimately familiar with.
If you can provide some further guidance I'd be happy to assist, but for the time being I'd like to sit back and see what @joshimoo was working on with their commit.
I do have a preliminary changeset for longhorn/longhorn-instance-manager that tweaks the nsenter stuff to use the system path instead of hardcoded paths - happy to push that if it would assist.

We currently do not have the capacity to test against non standard operating systems, for a list of our recommended operating systems have a look here: https://longhorn.io/docs/1.1.0/best-practices/#software
We do want to have another look at this issue once we are closer to the 1.2 release.

For folks using helmfiles you can use a patch to get it working.
For example:
repositories:
  - name: longhorn
    url: https://charts.longhorn.io
releases:
  - name: longhorn
    namespace: longhorn-system
    chart: longhorn/longhorn
    version: 1.3.2
    strategicMergePatches:
      - kind: DaemonSet
        apiVersion: apps/v1
        metadata:
          name: longhorn-manager
          namespace: longhorn-system
        spec:
          template:
            spec:
              containers:
                - name: longhorn-manager
                  env:
                    - name: PATH
                      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/run/wrappers/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin

Also, if you are like me, and you run the root filesystem out of tmpfs mount, you'll need to create and mount persistent storage for the default location /var/lib/longhorn.

@nigelgbanks a mutating admission webhook for pods in the longhorn-system namespace could be used a workaround in the mean time. To modify the pod spec as part of the creation api request.

@joshimoo thanks, for the recommendation, it was interesting reading. Shame PodPreset has been removed that would have worked a treat. Writing a webhook for this seems a bit excessive, and a bit much of a maintenance burden. I've decided to just maintain a custom image in the same way @duckfullstop has done.

Oh just found this https://artifacthub.io/packages/helm/kyverno/kyverno which might suffice.

Updated helm file example:
repositories:
  - name: longhorn
    url: https://charts.longhorn.io
  - name: kyverno
    url: https://kyverno.github.io/kyverno
  - name: incubator
    url: https://charts.helm.sh/incubator
releases:
  - name: longhorn
    namespace: longhorn-system
    chart: longhorn/longhorn
    version: 1.3.2
  - name: kyverno
    namespace: kyverno
    chart: kyverno/kyverno
    version: 2.6.1
  - name: longhorn-admission-hooks
    namespace: longhorn-system
    chart: incubator/raw
    needs:
      - kyverno/kyverno
    values:
      - resources:
          - apiVersion: v1
            kind: ConfigMap
            metadata:
              name: longhorn-custom-path
              namespace: longhorn-system
            data:
              PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/run/wrappers/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
          - apiVersion: kyverno.io/v1
            kind: ClusterPolicy
            metadata:
              name: add-host-path-to-longhorn
              annotations:
                policies.kyverno.io/title: Add Environment Variables from ConfigMap
                policies.kyverno.io/subject: Pod
                policies.kyverno.io/category: Other
                policies.kyverno.io/description: >-
                  Longhorn invokes executables on the host system, and needs
                  to be aware of the host systems PATH. This modifies all
                  deployments such that the PATH is explicitly set to support
                  NixOS based systems.
            spec:
              rules:
                - name: add-env-vars
                  match:
                    resources:
                      kinds:
                        - Pod
                      namespaces:
                        - longhorn-system
                  mutate:
                    patchStrategicMerge:
                      spec:
                        initContainers:
                          - (name): "*"
                            envFrom:
                              - configMapRef:
                                  name: longhorn-custom-path
                        containers:
                          - (name): "*"
                            envFrom:
                              - configMapRef:
                                  name: longhorn-custom-path

Does anyone have a working version of the above for kustomize? And of course having this issue fixed instead of workaround would be a major improvement

Instead of adding /run/wrappers/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin to PATH, it would be nice to be able to inject a custom binPath from the outside and actually get the Longhorn dependencies listed in the project README -  instead of magic installers like: https://github.com/longhorn/longhorn/blob/master/deploy/prerequisite/longhorn-cifs-installation.yaml#L8

failing to start iscsi i have openiscsi installed
 Failed to discover\" error=\"failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net iscsiadm -m disco │
│ very -t sendtargets -p <IP_ADDRESS>], output , stderr iscsiadm: can't open iscsid.startup configuration file etc/iscsi/iscsid.conf\\niscsiadm: iscsid is not running. Could not start it up automatically using the startup command in the iscsid.conf iscsid.startup setting. Please ch │
│ eck that the file exists or that your init scripts have started iscsid.\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: can't open iscsid.startup configuration file etc/iscsi/iscsid.conf\\niscsiadm: iscsid is not running. Could not start it up automatically using │
│  the startup command in the iscsid.conf iscsid.startup setting. Please check that the file exists or that your init scripts have started iscsid.\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could no │
│ t perform SendTargets discovery: could not connect to iscsid\\n: exit status 20\"

For me it was two things for nixos. The most important was @joaojacome 's input but I also was missing openiscsi on the nodes. So every node that is in my cluster now has this additional configuration
systemd.tmpfiles.rules = [
  "L+ /usr/local/bin - - - - /run/current-system/sw/bin/"
];

environment.systemPackages = with pkgs; [
  openiscsi
];

So far this seems to work. Will post updates if I discover more. But this also seems to point at @gnufied s question that nsenter is used for iscsiadm at least that was what my logs suggested as well

@bananflugan have you tried the kyverno yaml included here

@bananflugan have you tried the kyverno yaml included here

Yeah i installed Kyverno into its own namespace and then installed the policy resource into longhorn-system namespace. Sadly it did nothing for me, but i dont know if i did something wrong or missing something.

@bananflugan ensure you are awaiting I had to add these waits in before mine would work:
#!/bin/sh
kubectl_native_wait () {
  TARGET_POD=$1
  TARGET_NAMESPACE=$2
  kubectl wait --namespace $TARGET_NAMESPACE --for=condition=ready pod $TARGET_POD --timeout=120s
}

THIS_CWD=$(pwd)
set -eux

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.12.0/install.yaml
kubectl create ns longhorn-system
#kubectl label namespace longhorn-system istio-injection=enabled --overwrite
kubectl apply -f longhornfixnix.yml

kubectl_native_wait $(kubectl get po -n kyverno|grep kyverno-admission-controller|cut -f1 -d ' ') kyverno
kubectl_native_wait $(kubectl get po -n kyverno|grep kyverno-background-controller|cut -f1 -d ' ') kyverno
kubectl_native_wait $(kubectl get po -n kyverno|grep kyverno-cleanup-controller|cut -f1 -d ' ') kyverno
kubectl_native_wait $(kubectl get po -n kyverno|grep kyverno-reports-controller|cut -f1 -d ' ') kyverno
sleep 5

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

helm install longhorn \
  longhorn/longhorn \
  --namespace longhorn-system \
  --create-namespace \
  --version 1.6.2 \
  -f longhorn-values.yml

where that fixnix file is just the stuff from that wiki:
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: longhorn-nixos-path
  namespace: longhorn-system
data:
  PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/run/wrappers/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: longhorn-add-nixos-path
  annotations:
    policies.kyverno.io/title: Add Environment Variables from ConfigMap
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/category: Other
    policies.kyverno.io/description: >-
      Longhorn invokes executables on the host system, and needs
      to be aware of the host systems PATH. This modifies all
      deployments such that the PATH is explicitly set to support
      NixOS based systems.
spec:
  rules:
    - name: add-env-vars
      match:
        resources:
          kinds:
            - Pod
          namespaces:
            - longhorn-system
      mutate:
        patchStrategicMerge:
          spec:
            initContainers:
              - (name): "*"
                envFrom:
                  - configMapRef:
                      name: longhorn-nixos-path
            containers:
              - (name): "*"
                envFrom:
                  - configMapRef:
                      name: longhorn-nixos-path
---

and I most assuredly can bound a volume:
 k get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
csi-longhorn   Bound    pvc-b230f9d8-fe57-4cde-ae6a-aee942bdab46   4Gi        RWO            longhorn       4s

with pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: csi-longhorn
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

and test pod:
apiVersion: v1
kind: Pod
metadata:
  name: hello-local-csi-longhorn-pod
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: csi-longhorn
  containers:
  - name: hello-container
    image: busybox
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage

@Lindenk I created a helmfile that uses kustomize instead of kyverno to add the PATH environment variable:

helmfile.yaml
repositories:
  - name: longhorn
    url: https://charts.longhorn.io
  - name: incubator
    url: https://charts.helm.sh/incubator
releases:
  - name: longhorn
    namespace: longhorn-system
    chart: longhorn/longhorn
    version: 1.7.2
    needs:
      - longhorn-nixos
    jsonPatches:
      - target:
          kind: DaemonSet
          name: longhorn-manager
          namespace: longhorn-system
        patch:
          - op: add
            path: /spec/template/spec/containers/0/envFrom
            value:
              - configMapRef:
                name: longhorn-nixos-path
      - target:
          kind: Deployment
          name: longhorn-driver-deployer
          namespace: longhorn-system
        patch:
          - op: add
            path: /spec/template/spec/containers/0/envFrom
            value:
              - configMapRef:
                  name: longhorn-nixos-path
      - target:
          kind: Job
          namespace: longhorn-system
        patch:
          - op: add
            path: /spec/template/spec/containers/0/envFrom
            value:
              - configMapRef:
                  name: longhorn-nixos-path
  - name: longhorn-nixos
    namespace: longhorn-system
    chart: incubator/raw
    values:
      - resources:
          - apiVersion: v1
            kind: ConfigMap
            metadata:
              name: longhorn-nixos-path
              namespace: longhorn-system
            data:
              PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/run/wrappers/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin


google_sign_in

After i sign out, i try to log in again, yes it logs in succesfully but the problem is, i start to get error [cloud_firestore/permission-denied] but when i restart my application, it works again well. What am i missing?

Hi @AliKales
i start to get error [cloud_firestore/permission-denied] is a Flutterfire error
From what I can see, the issue is related to a FlutterFire plugin rather than to Flutter itself. Please open the issue in the dedicated repository.
Closing, as this isn't an issue with Flutter itself.
If you disagree, please write in the comments,  providing your flutter doctor -v, your pubspec.yaml, a minimal reproducible code sample that does not use 3rd party plugins, your complete flutter run -v logs running the minimal code sample
and I will reopen it.
Thank you


"No module files found" error when trying to generate components

Mac OSX 10.11.6
angular-cli: 1.0.0-beta.19-3
Node: 6.9.0
When developing apps created with the CLI, sometimes the ng g / ng generate command fails apparently arbitrarily. It'll be working fine and then suddenly stop working and throw an error. Sometimes the next time I come back to the project the following day, it will work again.
Example:
ng g component home
ng generate directive test
...etc.
The only output received whenever this happens is a red error stating:
Error locating module for declaration
        SilentError: No module files found

I've encountered this using CLI versions beta.18 as well as beta.19-3 and on multiple projects.

Same here, running windows 7

Same here:
Windows 10
angular-cli: 1.0.0-beta.19-3
Node 6.6.0

I've never seen it myself but the sporadic nature of it might indicate a race condition. @Brocco can you have a look?

I have almost the same issue:

Error locating module for declaration
SilentError: Multiple module files found:

May be the reason for that is changing some of the project file when adding router.

I have the same issue on osx.
If I try to create a component in the project root, I have the same error. If I use the same command in the directory where my project's module definition ts file is located, the cli generates the component. Though the location is relative to the module def file.

angular-cli: 1.0.0-beta.20-4
node: 6.9.1
os: linux x64
Running the command to generate new components gives an error and the components are not created.
example:
ng generate component edit-constant
This is just that it comes out after executing the command:

installing component
Error locating module for declaration
SilentError: No module files found

I think it's because people have moved the app.module.ts from the root of the app.
For instance when I moved my app.module.ts to a folder app/modules/ the error pops up.

@richardsengers  I confirm, it was the problem in my case. When I moved it to the root of the app, it worked fine.

This seems to be working for me too after updating my projects to move the app module file. Sorry for the pollution of the issue with references!

After I have tried on my own project,
I think in angular 2, if we create some file name like xxx.module.ts inside the app folder, this error occurs, if we remove this xxx.module.ts file inside src/app folder, we can generate the component files.
I have tried this solution and which works for me

In my case i happen to have another .angular-cli.json file in my src folder. Removing it solved the problem

Hi everyone;
In my case i use yeoman aspcore generator. And app.module.ts is different.  Later installations of .core seem to split the module into 3 files. client, server and shared.
if i try to install a new component "ng g component xForm" i take "specified module not found" error
Any idea?? :\

change app.module.shared.ts to app.module.ts and rename path in client and server module

You can create your component as follows  "ng g c home --module=app.module.ts" This worked for me :D


components: pagination

you should provide a pagination composant

Hey Augustin, actually there is a pagination component:
https://www.neobrutalism.dev/shadcn/components/pagination
I suppose you meant purely react component (without shadcn). In that case, I won't add it because there is no point in adding it (I would have to spend a lot of time making the component accessible and reusable, and still it wouldn't be good as radix-ui component), you can just use shadcn component and you're good.

Oh great, thanks. I hadn't realised there was a difference between pure react and shadcn.

and btw thanks for this project it's an awesome resssource


Remove global Fabrication.manager reference in Defintion

Opening this PR to see if you're open to me removing more global references inside of Fabrication.
I have a weird use case where I need to use Fabrication both for testing/development in rails, but I also need a completely separate object generator in production that will live in app/factories. I'm hoping to isolate all of the global/singleton state into one class that lives outside of the Manager for the "normal" test/development use case.

Sure, I am open to talking about it. Can you explain in more detail why exactly the singleton here is an issue for your development and test case?

Yeah!
I actually need to maintain two completely different sets of factories. The obvious set is for testing and development environments, which ‘Fabricator(:blah)’ handles well.
The problem is I don’t want the factories that will be used in my application code to have access to the test/development factories, so I need to create another instance of the manager to handle those factories.
From what I’ve seen so far in the code, most of the work involved in getting this out of a global dependency is the commits I’ve already done, then figuring out how to extract the file loader into another class that the manager can delegate to, and finally moving the ‘Factory(:blah)’ methods into a class, then delegating to an instance of that from the ‘Factory’ class/module

Would it not work for you to just override the fabricator path for production? Something like this:
Fabrication.configure do |config|
  config.fabricator_path = Rails.env.production? ? 'spec/fabricators' : 'app/fabricators'
end

It wouldn’t in this case because I need to test the models that user the factories from the production code.

On May 23, 2019 at 14:44,  <Paul Elliott (mailto:notifications@github.com)>  wrote:
Would it not work for you to just override the fabricator path for production? Something like this:
Fabrication.configure do |config| config.fabricator_path = Rails.env.production? ? 'spec/fabricators' : 'app/fabricators' end
—
You are receiving this because you authored the thread.
Reply to this email directly,   view it on GitHub (https://github.com/paulelliott/fabrication/pull/313?email_source=notifications&email_token=AAABEFDLK7D3UWQNIVWUGJTPW4F5XA5CNFSM4HO2N6LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDSLIY#issuecomment-495396259), or   mute the thread (https://github.com/notifications/unsubscribe-auth/AAABEFFSH3OE3SCAABRBUHDPW4F5XANCNFSM4HO2N6LA).

You could specify multiple paths. Like a base fabricator path then switch
on a second folder that has prod and no prod. Would that work?
I'm not opposed to these changes but if there's a simple way to do it
that's less risky I feel like it's worth exploring.
On Thu, May 23, 2019, 6:21 PM Brad Gessler<EMAIL_ADDRESS>wrote:

It wouldn’t in this case because I need to test the models that user the
factories from the production code.

On May 23, 2019 at 14:44, <Paul Elliott (mailto:notifications@github.com)>
wrote:
Would it not work for you to just override the fabricator path for
production? Something like this:
Fabrication.configure do |config| config.fabricator_path =
Rails.env.production? ? 'spec/fabricators' : 'app/fabricators' end
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub (
https://github.com/paulelliott/fabrication/pull/313?email_source=notifications&email_token=AAABEFDLK7D3UWQNIVWUGJTPW4F5XA5CNFSM4HO2N6LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDSLIY#issuecomment-495396259),
or mute the thread (
https://github.com/notifications/unsubscribe-auth/AAABEFFSH3OE3SCAABRBUHDPW4F5XANCNFSM4HO2N6LA
).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/paulelliott/fabrication/pull/313?email_source=notifications&email_token=AAAOJ5EGLE5EFVRV35TQNV3PW4KG5A5CNFSM4HO2N6LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDUW3Q#issuecomment-495405934,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAOJ5DX53RSPNAJYMLBBLDPW4KG5ANCNFSM4HO2N6LA
.


inscriptions.json updated Ordinal All-Stars

@MillHaus33

@MillHaus33

@MillHaus33 why do we still see the old inscriptions in your website but you have merged this new one and even in your code if you look in our collection there is the inscriptions.json file updated? Is there a problem?

@MillHaus33 we would like to drop our collection but if you don’t fix this problem, we can’t go further


AzureRm.ResourceGraph module not found in PSGallery

Running the command below fails to find the module in PSGallery
PS C:> Install-Module AzureRm.ResourceGraph

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d477e300-fccf-298b-fb97-d67d2f5acaec
Version Independent ID: 78f816e8-7ccf-9398-cbd7-940150f7d068
Content: Run your first Resource Graph query using Azure PowerShell
Content Source: articles/governance/resource-graph/first-query-powershell.md
Service: resource-graph
GitHub Login: @DCtheGeek
Microsoft Alias: dacoulte

@cbellee Thanks for letting me know.  The AzureRm.ResourceGraph mode was flagged as Prerelease, so it needs a flag to install: Install-Module -Name AzureRM.ResourceGraph -AllowPrerelease. I've already got the update to the doc staged and it should publish this morning.  Thanks!
@MicrosoftDocs/azure-cxp-triage Please label as 'doc-bug', assign to me, and #in-progress.

PS C:\windows\system32> Install-Module -Name AzureRM.ResourceGraph -AllowPrerelease
Install-Module : A parameter cannot be found that matches parameter name 'AllowPrerelease'.
At line:1 char:44

Install-Module -Name AzureRM.ResourceGraph -AllowPrerelease

                                       ~~~~~~~~~~~~~~~~

CategoryInfo          : InvalidArgument: (:) [Install-Module], ParameterBindingException
FullyQualifiedErrorId : NamedParameterNotFound,Install-Module

@MrCNeale What version is your PowerShellGet module? Get-Module -Name PowerShellGet. It looks like you have an older version that doesn't support the -AllowPrerelease flag and you'll need to update it. Please see https://docs.microsoft.com/powershell/gallery/installing-psget#get-the-latest-version-from-powershell-gallery

@DCtheGeek I can't find the update in the docs saying to use latest version of PowerShellGet. I ran into the same problem as OP with PowerShellGet version <IP_ADDRESS>.

@xenalite It can probably still be made clearer.  I'll work on improving it.  Thanks for the feedback!


[Bug] BD not launching with discord.

I just updated discord and BD didnt launch with it. I reinstalled BD at least 3 times and its still not working.

Duplicate of #685 & #688 ?