Set up connection using sc from sparkR

Is there a way to set up the sparklyr spark context using a pre-existing sc from sparkR?

+1 on this. The R session that I start already has a sc object initialized (I believe from some methods from sparkR, which are called to connect to a remote spark cluster). I think when you initialize a sc from sparklyr it creates a connection that is dplyr friendly. In line 70 of https://github.com/rstudio/sparklyr/blob/master/R/connection_spark.R
it looks like there is a function to autodetect if a sc already exists. Can sparklyr be initialized by passing in an already configured sc?

There is no way to share the spark connection between sparkR and sparklyr since two different backends manage this connection. If there is something in particular that you can do in sparkR but not in sparklyr, please open a github issue for us to take a look.

The spark environment I'm in won't load with sparklyr out of the box. If there's a good way to handle a custom environment I can make it easier for a lot of people.

@rgbkrk Yes, for sure. How does that custom environment looks like?

Would it help for me to upload a sanitized spark conf + environment variable setup?

@rgbkrk First, it would help to answer: Which operating system are you running this in? Was spark installed with spark_install? If not, which Spark distribution are you using? How does your master url connection look like? I can ask more questions once I have those answers. Cheers.

Sorry to take so long to answer this @javierluraschi. I wanted to make sure I understood what to report here.
We have an internal build of Spark that we backport fixes to. We run on yarn (master is always yarn-client). SPARK_HOME is set dynamically on run with some wrapper scripts.

@rgbkrk it really depends on the changes backported or performed in the custom environment. What would help me the most to troubleshoot this would be for you to share the error you get while running:
sc <- spark_connect(master = "<your-master>")
and I'll suggest something reasonable from there. However, it is definitely the that we won't be able to reuse a connection that was created using sparkR.init().

Thanks @javierluraschi. Would you be alright with me sending you the traceback by private email?

I snipped a bit of it (I'd love to provide more context, apologies here):
> spark_connect(master='yarn-client')
Error in force(code) :
  Failed while connecting to sparklyr to port (8880) for sessionid (2708): Gateway in port (8880) did not respond.
    Path: <PATH>/spark-1.6.1/bin/spark-submit
    Parameters: --class, sparklyr.Backend, --packages, 'com.databricks:spark-csv_2.11:1.3.0', '/usr/local/lib/R/site-library/sparkly
r/java/sparklyr-1.6-2.10.jar', 8880, 2708
    Traceback:
      shell_connection(master = master, spark_home = spark_home, app_name = app_name, version = version, hadoop_version = hadoop_ver
sion, shell_args = shell_args, config = config, service = FALSE, extensions = extensions)
      start_shell(master = master, spark_home = spark_home, spark_version = version, app_name = app_name, config = config, jars = sp
ark_config_value(config, "spark.jars.default", list()), packages = spark_config_value(config, "sparklyr.defaultPackages"), extension
s = extensions, environment = environment, shell_args = shell_args, service = service)
      tryCatch({
    gatewayInfo <- spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, config = config, isStarting = TRUE)
}, error = function(e) {
    abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$message
, sep = ""), spark_submit_path, shell_args, output_file, error_file)
})
      tryCatchList(expr, classes, parentenv, handlers)
      tryCatchOne(expr, names, parentenv, handlers[[1]])
      value[[3]](cond)
      abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$messa
ge, sep = ""), spark_submit_path, shell_args, output_file, error_file)

---- Output Log ----
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
        at org.apache.spark.deploy.SparkSubmitArguments.handleUnknown(SparkSubmitArguments.scala:451)
        at org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:178)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:97)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 5 more

---- Error Log ----

One setting that I think we'll want to propagate is that we need deploy-mode set to client on spark-submit:
If I had to guess, this would shape the sparklyr arguments to spark-submit to be:
spark-submit --deploy-mode client --class sparklyr.Backend --packages 'com.databricks:spark-csv_2.11:1.3.0 /usr/local/lib/R/site-library/sparklyr/java/sparklyr-1.6-2.10.jar 8880 2708

It's also highly likely that we should be building sparklyr against our current version of Scala + Spark.
/cc @rdblue

@rgbkrk in order to set --deploy-mode, try:
config <- spark_config()
config[["sparklyr.shell.deploy-mode"]] <- "client"
sc <- spark_connect(master = "", config = config)

Is this sufficient?

> sc <- spark_connect(master="yarn-client", config=config)
Error in force(code) :
  Failed while connecting to sparklyr to port (8880) for sessionid (6299): Gateway in port (8880) did not respond.
    Path: /tmp/bdp-clients/root/20161027_002032.009726.prodsparkshell161/genie/applications/spark161/dependencies/spark-1.6.1/bin/sp
ark-submit
    Parameters: --class, sparklyr.Backend, --packages, 'com.databricks:spark-csv_2.11:1.3.0', '/usr/local/lib/R/site-library/sparkly
r/java/sparklyr-1.6-2.10.jar', 8880, 6299
    Traceback:
      shell_connection(master = master, spark_home = spark_home, app_name = app_name, version = version, hadoop_version = hadoop_ver
sion, shell_args = shell_args, config = config, service = FALSE, extensions = extensions)
      start_shell(master = master, spark_home = spark_home, spark_version = version, app_name = app_name, config = config, jars = sp
ark_config_value(config, "spark.jars.default", list()), packages = spark_config_value(config, "sparklyr.defaultPackages"), extension
s = extensions, environment = environment, shell_args = shell_args, service = service)
      tryCatch({
    gatewayInfo <- spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, config = config, isStarting = TRUE)
}, error = function(e) {
    abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$message
, sep = ""), spark_submit_path, shell_args, output_file, error_file)
})
      tryCatchList(expr, classes, parentenv, handlers)
      tryCatchOne(expr, names, parentenv, handlers[[1]])
      value[[3]](cond)
      abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$messa
ge, sep = ""), spark_submit_path, shell_args, output_file, error_file)

---- Output Log ----
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
        at org.apache.spark.deploy.SparkSubmitArguments.handleUnknown(SparkSubmitArguments.scala:451)
        at org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:178)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:97)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 5 more

---- Error Log ----

Looking at config, it appears that sparklyr is not picking up all my environment variables.

> config
$sparklyr.cores.local
[1] 32

$spark.sql.shuffle.partitions.local
[1] 32

$spark.env.SPARK_LOCAL_IP.local
[1] "<IP_ADDRESS>"

$sparklyr.defaultPackages
[1] "com.databricks:spark-csv_2.11:1.3.0"

$`sparklyr.shell.deploy-mode`
[1] "client"

attr(,"config")
[1] "default"
attr(,"file")
[1] "/usr/local/lib/R/site-library/sparklyr/conf/config-template.yml"

@rgbkrk Yes, there was a regression reading values from the config file that got fixed with commit (https://github.com/rstudio/sparklyr/commit/355d62828be2e6fdf929b228c9c5ff608a0385d9), please use sparklyr 0.4.21 or newer to address this issue.

> sc <- spark_connect(master="yarn-client", config=config)
Error in force(code) :
  Failed while connecting to sparklyr to port (8880) for sessionid (3228): Gateway in port (8880) did not respond.
    Path: /tmp/bdp-clients/root/20161028_181705.059266.prodsparkshell161/genie/applications/spark161/dependencies/spark-1.6.1/bin/sp
ark-submit
    Parameters: --deploy-mode, client, --class, sparklyr.Backend, --packages, 'com.databricks:spark-csv_2.11:1.3.0', '/usr/local/lib
/R/site-library/sparklyr/java/sparklyr-1.6-2.10.jar', 8880, 3228
    Traceback:
      shell_connection(master = master, spark_home = spark_home, app_name = app_name, version = version, hadoop_version = hadoop_ver
sion, shell_args = shell_args, config = config, service = FALSE, extensions = extensions)
      start_shell(master = master, spark_home = spark_home, spark_version = version, app_name = app_name, config = config, jars = sp
ark_config_value(config, "spark.jars.default", list()), packages = spark_config_value(config, "sparklyr.defaultPackages"), extension
s = extensions, environment = environment, shell_args = shell_args, service = service)
      tryCatch({
    gatewayInfo <- spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, config = config, isStarting = TRUE)
}, error = function(e) {
    abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$message
, sep = ""), spark_submit_path, shell_args, output_file, error_file)
})
      tryCatchList(expr, classes, parentenv, handlers)
      tryCatchOne(expr, names, parentenv, handlers[[1]])
      value[[3]](cond)
      abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$messa
ge, sep = ""), spark_submit_path, shell_args, output_file, error_file)

---- Output Log ----
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
        at org.apache.spark.deploy.SparkSubmitArguments.handleUnknown(SparkSubmitArguments.scala:451)
        at org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:178)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:97)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 5 more

---- Error Log ----

It looks like there is more for me to add on to the config, though I'm not sure what sparklyr expects. I do have environment variables set from a sparkR session.
> config
$sparklyr.cores.local
[1] 32

$spark.sql.shuffle.partitions.local
[1] 32

$spark.env.SPARK_LOCAL_IP.local
[1] "<IP_ADDRESS>"

$sparklyr.defaultPackages
[1] "com.databricks:spark-csv_2.11:1.3.0"

$`sparklyr.shell.deploy-mode`
[1] "client"

attr(,"config")
[1] "default"
attr(,"file")
[1] "/usr/local/lib/R/site-library/sparklyr/conf/config-template.yml"

By the way, I'd love to pay for consulting work to help with integration work in our environment for sparklyr with the intent that it helps the open source project and all users in general. I'm reachable at<EMAIL_ADDRESS>and<EMAIL_ADDRESS>

@rgbkrk I'll forward your contact within my team to reach out. Could you share sessionInfo()?

> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.5 LTS

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] sparklyr_0.4.21 SparkR_1.6.1

loaded via a namespace (and not attached):
 [1] magrittr_1.5    R6_2.2.0        assertthat_0.1  rprojroot_1.0-2
 [5] parallel_3.3.1  tools_3.3.1     DBI_0.5-1       withr_1.0.2
 [9] dplyr_0.5.0     yaml_2.1.13     tibble_1.2      Rcpp_0.12.7
[13] config_0.2      digest_0.6.10

I'm using sparklyr straight from github master here by the way.

@rgbkrk oh right! Yes, I missed this in your logs: Parameters: --deploy-mode, client, --class, sparklyr.Backend, --packages, 'com.databricks:spark-csv_2.11:1.3.0', '/usr/local/lib /R/site-library/sparklyr/java/sparklyr-1.6-2.10.jar', 8880, 3228 which means progress since at least deploy-mode is being set to client.
So now we need to focus on this issue:
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream

Related: http://stackoverflow.com/questions/30906412/noclassdeffounderror-com-apache-hadoop-fs-fsdatainputstream-when-execute-spark-s
Could you talk a bit about your cluster? Which distribution of Spark is being used? If this is CDH we should point Sys.setEnv(SPARK_HOME = "") to the CDH Spark directory. Did you run spark_install for this client?

We run our own distribution of Spark. We run on top of EMR and don't use Cloudera or Hortonworks.

> Sys.getenv('SPARK_HOME')
[1] "/tmp/bdp-clients/root/20161028_181705.059266.prodsparkshell161/genie/applications/spark161/dependencies/spark-1.6.1"
> Sys.getenv('HADOOP_HOME')
[1] "/tmp/bdp-clients/root/20161028_181705.059266.prodsparkshell161/genie/applications/hadoop272/dependencies/hadoop-2.7.2"

/cc @rdblue

@rgbkrk Let's try to remove databricks csv package from your config object:
config$sparklyr.defaultPackages <- list()
sc <- spark_connect(master="yarn-client", config=config)

Someone is referencing the hadoop binaries, sparklyr does not really reference hadoop at all but is possible that the databricks csv module does.

Same issue. Should I clear out my environment variables for SPARK_DRIVER_CLASSPATH and SPARK_SUBMIT_ARGS?

@rgbkrk That could help pinpoint this issue, let us know how it goes! Also, if you were to find somewhere in your logs the full call stack from java.lang.NoClassDefFoundError that would also help.

@rgbkrk @javierluraschi
This is the code I used to get the connection with a Cloudera cluster to work.  I had to update the Hadoop and Yarn configuration environment variables to the right file.  I hoping that maybe some of the custom setup your Hadoop cluster has may be similar to Cloudera's:
library(sparklyr)
library(dplyr)
Sys.setenv(HADOOP_CONF_DIR='/etc/hadoop/conf.cloudera.hdfs')
Sys.setenv(YARN_CONF_DIR='/etc/hadoop/conf.cloudera.yarn')
sc <- spark_connect(master = "yarn-client", version="1.6.0", spark_home = '/opt/cloudera/parcels/CDH/lib/spark/')

Thanks @edgararuiz. The problem here is a little different because the Spark classpath is made from two components in our setup: the Hadoop classpath and Spark classpath (directory). Cloudera is a bit easier because their directory setup has all dependencies of an application like Spark linked in the app's lib folder.

Thanks @rdblue .  Quick question, is the data you're trying to analyze in Hive? And if so, what version and distribution of Hadoop are you running?

@rgbkrk sparklyr does not determine the java path and does not require a java path. sparklyr has a dependency to spark-submit, so as long as the spark-submit script can read the right java path and submit the jar appropiately to Spark that's all its expected.
From the error, it looks to me like SPARK_DIST_CLASSPATH might not be set and this is causing Spark to not launch spark-submit, or at least, for standard Spark distributions, this would be the case.
@rgbkrk sparklyr is not expected to pick any environment variables, instead, sparklyr relies in spark-submit which does read environment variables but there shouldn't be a need to proxy them.
@rdblue @rgbkrk what would happen if the SimpleApp jar from http://spark.apache.org/docs/latest/quick-start.html#self-contained-applications is submitted to your Spark distribution through spark-submit? Would this work? Assuming it works, what would happen if you submit this application through R with something like:
system2(paste(
  "YOUR_SPARK_HOME/bin/spark-submit",
  "--class SimpleApp",
  "--master local[4]",
  "target/scala-2.11/simple-project_2.11-1.0.jar"))

could you confirm this would also work as expected?
Side Node: I was able to get this same exception while running a without-hadoop distribution of Spark and not setting an environment variable for Hadoop, this was the case even for the SimpleApp jar so it seems likely that in order to resolve this issue, a global environment to the hadoop binaries will be needed.

Guys I have spent hours trying to figure out the issue klyr Error in force(code) : Failed while connecting to sparklyr to port (8880) for sessionid (792): Gateway in port (8880) did not respond. Path: /home/ubuntu/spark-1.6.2-bin-hadoop2.6/bin/spark-submit Parameters: --class, sparklyr.Backend, --packages, 'ai.h2o:sparkling-water-core_2.10:1.6.7','ai.h2o:sparkling-water-ml_2.10:1.6.7','ai.h2o:sparkling-water-repl_2.10:1.6.7', '/usr/local/lib/R/site-library/sparklyr/java/sparklyr-1.6-2.10.jar', 8880, 792 Traceback: shell_connection(master = master, spark_home = spark_home, app_name = app_name, version = version, hadoop_version = hadoop_version, shell_args = shell_args, config = config, service = FALSE, extensions = extensions) start_shell(master = master, spark_home = spark_home, spark_version = version, app_name = app_name, config = config, jars = spark_config_value(config, "spark.jars.default", list()), packages = spark_config_value(config, "sparklyr.defaultPackages"), extensions = extensions, environment = environment, shell_args = shell_args, service = service) tryCatch({ gatewayInfo <- spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, config = config, isStarting = TRUE) }, error = function(e) { abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$message, sep = ""), spark_submit_path, shell_args, output_file, error_file) }) tryCatchList(expr, classes, parentenv, handlers) tryCatchOne(expr, names, parentenv, handlers[[1]]) value[3] abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$message, sep = ""), spark_submit_path, shell_args, output_file, error_file) ---- Output Log ---- Ivy Default Cache set to: /var/www/.ivy2/cache The jars for the packages stored in: /var/www/.ivy2/jars :: loading settings :: url = jar:file:/home/ubuntu/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml ai.h2o#sparkling-water-core_2.10 added as a dependency ai.h2o#sparkling-water-ml_2.10 added as a dependency ai.h2o#sparkling-water-repl_2.10 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] Exception in thread "main" java.io.FileNotFoundException: /var/www/.ivy2/cache/resolved-org.apache.spark-spark-submit-parent-1.0.xml (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream.(FileOutputStream.java:162) at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:70) at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:62) at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.toIvyFile(DefaultModuleDescriptor.java:563) at org.apache.ivy.core.cache.DefaultResolutionCacheManager.saveResolvedModuleDescriptor(DefaultResolutionCacheManager.java:176) at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:245) at org.apache.ivy.Ivy.resolve(Ivy.java:523) at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1066) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ---- Error Log ---- Calls: spark_connect ... tryCatchOne -> -> abort_shell -> -> force Execution halted

@edgararuiz, out clusters run EMR with Hadoop version 2.7.2. We have a Hive metastore and have profiles to run Hive 1.0, Hive 2.1.0, and SparkSQL based on Hive 1.2.1. Our setup is a little special in that we package applications to run through Genie, which is why we don't have a "normal" Hadoop setup. I think as long as we can pass environment variables and select the Spark startup script, we should be able to make this work, though.

@raghav20 the issue you mention is different than this one, please open a new github issue and I'll take a look there.

Hi Javier,
I am getting the same error. but the output log is different. I am using the most updated package of sparklyr mentioned in the github. I am getting the same error using Rstudio as well as R gui. Please help me to resolve the issue.
sc <- spark_connect(master = "local",config = config)
Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (2044): Gateway in port (8880) did not respond.
Path: C:\Users\user1\AppData\Local\rstudio\spark\Cache\spark-1.6.2-bin-hadoop2.6\bin\spark-submit2.cmd
Parameters: --class, sparklyr.Backend, --packages, "com.databricks:spark-csv_2.11:1.3.0", "D:\Users\user1\R\R-3.3.1\library\sparklyr\java\sparklyr-1.6-2.10.jar", 8880, 2044
Traceback:
shell_connection(master = master, spark_home = spark_home, app_name = app_name, version = version, hadoop_version = hadoop_version, shell_args = shell_args, config = config, service = FALSE, extensions = extensions)
start_shell(master = master, spark_home = spark_home, spark_version = version, app_name = app_name, config = config, jars = spark_config_value(config, "spark.jars.default", list()), packages = spark_config_value(config, "sparklyr.defaultPackages"), extensions = extensions, environment = environment, shell_args = shell_args, service = service)
tryCatch({
gatewayInfo <- spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, config = config, isStarting = TRUE)
}, error = function(e) {
abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$message, sep = ""), spark_submit_path, shell_args, output_file, error_file)
})
tryCatchList(expr, classes, parentenv, handlers)
tryCatchOne(expr, names, parentenv, handlers[[1]])
value[3]
abort_shell(paste("Failed while connecting to sparklyr to port (", gatewayPort, ") for sessionid (", sessionId, "): ", e$message, sep = ""), spark_submit_path, shell_args, output_file, error_file)
---- Output Log ----
The system cannot find the path specified.
---- Error Log ----
Session info:
R version 3.3.1 (2016-06-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server >= 2012 x64 (build 9200)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] sparklyr_0.4.21
loaded via a namespace (and not attached):
[1] Rcpp_0.12.7 digest_0.6.10 dplyr_0.5.0 withr_1.0.2 rprojroot_1.1 assertthat_0.1 rappdirs_0.3.1 R6_2.2.0 DBI_0.5-1 backports_1.0.4 magrittr_1.5 devtools_1.12.0
[13] config_0.2 tools_3.3.1 parallel_3.3.1 yaml_2.1.13 memoise_1.0.0 tibble_1.2

Is there a way to tell sparklyr the class path before it submits? I have SPARK_DRIVER_CLASSPATH set, sparklyr doesn't seem to pick it up.

@javierluraschi  I have the same problem. I have Spark 2.1.0 installed and I get an error
`> sc <- spark_connect(master="spark://local:6324", spark_home="/opt/spark-2.1.0/", version="2.1.0",hadoop_version="2.7")
Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (77): Gateway in port (8880) did not respond.
Path: /opt/spark-2.1.0/bin/spark-submit
Parameters: --class, sparklyr.Backend, --packages, 'datastax:spark-cassandra-connector:2.0.1-s_2.11', '/home/taras/R/x86_64-pc-linux-gnu-library/3.3/sparklyr/java/sparklyr-2.1-2.11.jar', 8880, 77
---- Output Log ----
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveContext
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:722)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveContext
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
---- Error Log ----
`
The SimpleApp and system(...)  both work properly.
I followed your remark, that you reproduced that problem in "without-hadoop" installation. Therefore, I installed Hadoop2.7, created "HADOOP-HOME" variables, but I still get the similar error but with another OutputLog
`> sc <- spark_connect(master="spark://local:6324", spark_home="/opt/spark-2.1.0/", version="2.1.0",hadoop_version="2.7")
Fehler in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (77): Gateway in port (8880) did not respond.
Path: /opt/spark-2.1.0/bin/spark-submit
Parameters: --class, sparklyr.Backend, --packages, 'datastax:spark-cassandra-connector:2.0.1-s_2.11', '/home/taras/R/x86_64-pc-linux-gnu-library/3.3/sparklyr/java/sparklyr-2.1-2.11.jar', 8880, 77
---- Output Log ----
Ivy Default Cache set to: /home/taras/.ivy2/cache
The jars for the packages stored in: /home/taras/.ivy2/jars
:: loading settings :: url = jar:file:/opt/spark-2.1.0/assembly/target/scala-2.11/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
datastax#spark-cassandra-connector added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found datastax#spark-cassandra-connector;2.0.1-s_2.11 in spark-packages
found joda-time#joda-time;2.3 in central
found io.netty#netty-all;4.0.33.Final in central
found commons-beanutils#commons-beanutils;1.9.3 in central
found commons-collections#commons-collections;3.2.2 in local-m2-cache
found org.joda#joda-convert;1.2 in central
found com.twitter#jsr166e;1.1.0 in central
found org.scala-lang#scala-reflect;2.11.8 in central
:: resolution report :: resolve 876ms :: artifacts dl 13ms
:: modules in use:
com.twitter#jsr166e;1.1.0 from central in [default]
commons-beanutils#commons-beanutils;1.9.3 from central in [default]
commons-collections#commons-collections;3.2.2 from local-m2-cache in [default]
datastax#spark-cassandra-connector;2.0.1-s_2.11 from spark-packages in [default]
io.netty#netty-all;4.0.33.Final from central in [default]
joda-time#joda-time;2.3 from central in [default]
org.joda#joda-convert;1.2 from central in [default]
org.scala-lang#scala-reflect;2.11.8 from central in [default]
---------------------------------------------------------------------
|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 8 already retrieved (0kB/19ms)
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveContext
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:722)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveContext
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
---- Error Log ----`
Do you have any clue how can I get it running?
P.S. It works with spark-install() version of spark, but not with a pre-installed one.
sparklyr_0.5.5
RStudio-server
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS

@lazariv is your spark up and running in standalone mode. If not then run that using the following commands.
go to SPARK_HOME/bin
spark-class.cmd org.apache.spark.deploy.master.Master
go to the path go to SPARK_HOME/bin again from another window and run the worker where spark master is running. spark://...:7077
spark-class2.cmd org.apache.spark.deploy.worker.Worker spark://...:7077
library(sparklyr)
sc <- spark_connect(master = "spark://...:7077",app_name = "test_spark")
dat <- copy_to(sc, iris, overwrite = TRUE)
Hope its helpful

@dsblr Thanks for your help, but unfortunately it didn't help.  I have the spark master and slave running.
But I started it again, following your instructions (I used spark-class since I am on Ubuntu), I have master running with:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/06/02 06:22:45 INFO Master: Started daemon with process name: 22827@rtest
17/06/02 06:22:45 INFO SignalUtils: Registered signal handler for TERM
17/06/02 06:22:45 INFO SignalUtils: Registered signal handler for HUP
17/06/02 06:22:45 INFO SignalUtils: Registered signal handler for INT
17/06/02 06:22:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/06/02 06:22:46 INFO SecurityManager: Changing view acls to: taras
17/06/02 06:22:46 INFO SecurityManager: Changing modify acls to: taras
17/06/02 06:22:46 INFO SecurityManager: Changing view acls groups to:
17/06/02 06:22:46 INFO SecurityManager: Changing modify acls groups to:
17/06/02 06:22:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(taras); groups with view permissions: Set(); users  with modify permissions: Set(taras); groups with modify permissions: Set()
17/06/02 06:22:46 WARN Utils: Service 'sparkMaster' could not bind on port 6324. Attempting port 6325.
17/06/02 06:22:46 INFO Utils: Successfully started service 'sparkMaster' on port 6325.
17/06/02 06:22:46 INFO Master: Starting Spark master at spark://<IP_ADDRESS>:6325
17/06/02 06:22:47 INFO Master: Running Spark version 2.1.0
17/06/02 06:22:47 WARN Utils: Service 'MasterUI' could not bind on port 8882. Attempting port 8883.
17/06/02 06:22:47 INFO Utils: Successfully started service 'MasterUI' on port 8883.
17/06/02 06:22:47 INFO MasterWebUI: Bound MasterWebUI to <IP_ADDRESS>, and started at http://<IP_ADDRESS>:8883
17/06/02 06:22:47 WARN Utils: Service could not bind on port 6066. Attempting port 6067.
17/06/02 06:22:47 INFO Utils: Successfully started service on port 6067.
17/06/02 06:22:47 INFO StandaloneRestServer: Started REST server for submitting applications on port 6067
17/06/02 06:22:47 INFO Master: I have been elected leader! New state: ALIVE

but I get the same error
Error in force(code) :
  Failed while connecting to sparklyr to port (8880) for sessionid (9582): Gateway in port (8880) did not respond.
    Path: /opt/spark-2.1.0/bin/spark-submit
    Parameters: --class, sparklyr.Backend, --packages, 'datastax:spark-cassandra-connector:2.0.1-s_2.11', '/home/taras/R/x86_64-pc-linux-gnu-library/3.3/sparklyr/java/sparklyr-2.1-2.11.jar', 8880, 9582

I tried modifying config.yaml with different ports (8883, 6325, 6067) like sparklyr.gateway.port: 8883 , but nothing helps. Only with 8883 I get some responses from spark-master process:
17/06/02 06:33:17 WARN HttpParser: Illegal character 0x0 in state=START for buffer HeapByteBuffer@2728ce84[p=1,l=12,c=16384,r=11]={\x00<<<\x00\x00\x00\x00\x00\x1c\xCa\x00\x00\x00\x00>>>x\xD9\x01"\xDb\x00\x00\x04\xFe\xAc\xEd\x00\x05sr\x00)...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
17/06/02 06:33:17 WARN HttpParser: badMessage: 400 Illegal character 0x0 for HttpChannelOverHttp@53d2403d{r=0,c=false,a=IDLE,uri=}
17/06/02 06:33:20 WARN HttpParser: Illegal character 0x0 in state=START for buffer HeapByteBuffer@2728ce84[p=1,l=12,c=16384,r=11]={\x00<<<\x00\x00\x00\x00\x00\x1c\xCa\x00\x00\x00<>>>x\xD9\x01"\xDb\x00\x00\x04\xFe\xAc\xEd\x00\x05sr\x00)...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
17/06/02 06:33:20 WARN HttpParser: badMessage: 400 Illegal character 0x0 for HttpChannelOverHttp@3c2c769b{r=0,c=false,a=IDLE,uri=}

Maybe there is another solution?

Touching base with @rgbkrk by mail, closing this one.

Sounds good to me.

I am getting the same error as @lazariv : Any solution?
sc <- spark_connect(master = "spark://<cluster/node>:7077")
Error in force(code) :
Failed during initialize_connection: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveContext
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
Output log:
18/05/22 14:59:13 ERROR sparklyr: Backend (39047) failed calling getSparkContext on 0 with no class found error


Hang when using ext-uv

@trowski @kelunik encountering a weird hang when using amphp/parallel and (only when using) ext-uv without opcache, the following example gets stuck with 100% CPU usage on ubuntu 20.04 with the ondrej ppa repos:
test.php:
<?php

use Amp\Parallel\Worker\ContextWorkerPool;
use danog\PixyReproducer\TestTask;
use Revolt\EventLoop;

use function Amp\Future\await;

require 'vendor/autoload.php';

$pool = new ContextWorkerPool(1000);

$states = [];
$futures = [];
for ($x = 0; $x < ($argv[1] ?? 10); $x++) {
    echo "Starting $x...\n";
    $task = $pool->submit(new TestTask);
    $futures []= $task->getFuture();

    $channel = $task->getChannel();
    EventLoop::queue(function () use ($channel, &$states, $x) {
        while ($data = $channel->receive()) {
            if (isset($data['ok'])) {
                $states[$x] = $data['ok'];
            } else {
                var_dump($data);
            }
        }
    });
}

echo "Done!\n";

await($futures);

TestTask.php:
<?php

namespace danog\PixyReproducer;

use Amp\Cancellation;
use Amp\Parallel\Worker\Task;
use Amp\Sync\Channel;

use function Amp\delay;

class TestTask implements Task {

    /**
     * Executed when running the Task in a worker.
     *
     * @param Channel<TReceive, TSend> $channel Communication channel to parent process.
     * @param Cancellation $cancellation Tasks may safely ignore this parameter if they are not cancellable.
     *
     * @return TResult A specific type can (and should) be declared in implementing classes.
     */
    public function run(Channel $channel, Cancellation $cancellation): mixed
    {
        delay(100);
    }
}

Result:
Starting 1...
Starting 2...
Starting 3...
Starting 4...
Starting 5...
Starting 6...
Starting 7...
Starting 8...
Starting 9...
Starting 10...
Starting 11...
Starting 12...
Starting 13...
Starting 14...
Starting 15...
Starting 16...
Starting 17...
Starting 18...
^C <stuck>

The same example works fine on my arch machine for some reason.

Also, the issue does not appear when starting say 15 workers instead of 20

oh nevermind, it's yet another JIT issue :(

Closing, as this issue appears to have been fixed in php-src.


Evaluate LN proposal

https://github.com/bithyve/internal-docs/issues/188

Evaluated and provided feedback to @antuz123


RTSP connection check

Hi guys,
Does anybody know how to check the status of the RTSP connection?
I tested streaming live video transmission to Wowza Server based on Example3.
But sometimes the server created broken mp4 files.
According to several test results, I believe bad network status cause this situation.
I guess that the transfer is started before the RTSP connection has been completed.
Even though I used TCP method, the result was same.
However there were no errors occurred, when I made a device waited for 3 seconds and then sent live video after it connected RTSP connection.
But in this way I’m not sure whether the connection is opened or not.
I wonder how I can check the status of the RTSP connection.
Thanks

You can send a "OPTION" to server
then the server will give you a code


Add support for storing attestations in oci/layout

This doesn't save empty signatures/attestations to disk, since we can't write them later anyway.
Also rename the annotation used to differentiate images to "kind"
Signed-off-by: Priya Wadhwa<EMAIL_ADDRESS>ref #1015
Release Note

Add support for storing attestations in `oci/layout`

good soup


Disable active effect on donut chart does not work

Description
I want to disable disable the active state of the donut chart but its not working. After clicking on the one of data label the size of the arc changes and it shows as a active. I am using state option to disable this behaviour. but its not working. could someone help me if I am doing something wrong here.
Steps to Reproduce
Clicking on the one of data label
Expected Behavior
I am using state option to disable the above behaviour. but its not working.
Actual Behavior
it does not even after the setting the state.active.filter: none.
Reproduction Link
https://codepen.io/apexcharts/pen/aaOrjE

Not a bug. I found an option to to disable the selected label by using property: expandOnClick: false,


Update Joseph Koshakow in people.md

Move Joseph Koshakow from Contributors to PMC/Committers

@jkosh44 you can link your apache and github accounts at https://gitbox.apache.org/.  After that you should be able to commit this.  Let me know if you run into any problems.


chore: added install path and fixed import on plugin-octopus-deploy

Hey, I just made a Pull Request!
added install path and fixed import on plugin-octopus-deploy readme
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[x] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Nice thanks! :tada:


validate and document how to debug on WSL2 windows

I'm not in a rush to get this working on windows native, but to how to:

get k3d running
figure out network routing
create commands for it.

I was able to try this out on a clean image using the following. Seems to work. Maybe there is a better way to get docker started. This seems to work for me though. Everything is pretty normal except a step to start docker.
Install docker engine
sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Change to iptables legacy to start docker
sudo update-alternatives --config iptables

# Enter 1 to select iptables-legacy

sudo service docker start

Install k3d and kubectl
curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash

k3d cluster create test-cluster

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

kubectl version --client

kubectl config use-context k3d-test-cluster\

Install  nodejs and pepr
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt install -y nodejs
node --version

npm i -g pepr

pepr init

cd pepr-helloworld
npm run k3d-setup

kubectl cluster-info

Open new terminal and apply hello world example
cd  pepr-helloworld
kubectl apply -f capabilities/hello-pepr.samples.yaml

namespace/pepr-demo created
configmap/example-1 created
configmap/example-2 created
configmap/example-3 created
configmap/example-4 created
configmap/example-5 created
customresourcedefinition.apiextensions.k8s.io/unicorns.pepr.dev created

Verify that they are created
kubectl get configmaps -n pepr-demo

NAME               DATA   AGE
kube-root-ca.crt   1      106s
example-1          1      106s
example-2          1      106s
example-3          2      106s
example-4          1      106s
example-5          2      105s

will update, it's actually fairly simple:
https://code.visualstudio.com/blogs/2019/09/03/wsl2
in order for the webhook to be forwarded, you need to open up port 3000. (maybe port 443 too).
https://gist.github.com/xmeng1/aae4b223e9ccc089911ee764928f5486
should do a full walk through of wsl2 from scratch with docker desktop.

Do not run Docker desktop, deploy dockerd directly in wsl2, docker desktop's WSL2 mode does not allow bridge networks to be easily accessible from wsl2 or windows. The metallb stuff will just not run well.


Xtrans sys/stropts.h fix for Fedora

Contributing to homebrew-xorg:
You can erase any parts of this template not applicable to your Pull Request.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[ ] Have you included a log of the failed build without your patch using brew gist-logs <formula>?
[x] Does your submission pass
brew audit --strict <formula> (where <formula> is the name of the formula you're submitting)?
[x] Have you built your formula locally prior to submission with brew install <formula>?
[x] Ensure that libx11 compiles successfully
[ ] Clean commit history
[x] Only apply patch for Fedora

Still needs testing, and to switch the URL for the patch

From the CircleCI log
==> Downloading https://raw.githubusercontent.com/richardcq/homebrew-xorg/xtrans_sys_stropts_h/Patches/patch_xtrans_ioctl.diff
Error: An exception occurred within a child process:
  ChecksumMismatchError: SHA256 mismatch
Expected: 195a6026ffb9c6b74ee132fc73b18ec03e149bdeffc6082c661723eb8b305309
Actual: 697366c6344c90a241600294064de595ea45cc417e7d5279ad79a9cd3e5355bf
Archive: /home/linuxbrew/.cache/Homebrew/downloads/762b09c459c9c4ea7954ccfeb8f67aab70c4e15fb0daebb4b501d97eafe9502e--patch_xtrans_ioctl.diff
To retry an incomplete download, remove the file above.

This isn't the sha256 I calculate using sha256sum patch_xtrans_ioctl.diff

@maxim-belkin this is ready for review, I tested the build on Fedora 28 and CentOS 7.6 to ensure the patch only applied on Fedora installs

Thanks, @richardcq!
Homebrew/Linuxbrew is usually hesitant to add distribution-specific changes to the formulae so I'll let @sjackman to chime in.

Assuming this patch doesn't break the build on non-Fedora systems, let's apply it unconditionally.

Thanks for this work, Richard!

@sjackman Assuming the travis build doesn't fail, this should be good to go

@sjackman sorry, I fixed the spacing while you approved that

No worries at all.

If you know how to squash these commits down to a single commit using git rebase -i origin/master, please do. Don't worry about it if not. We'll squash the commits when we merge the PR. For future PRs, you can use git commit --amend and git push --force to modify your previous commit without creating a new one.

Thank you for your first contribution to Linuxbrew/xorg, Richard! 🏅

I think we need a new bottle here...

No, my guess would be that this change does not affect the resulting binary.

I suggest squash-merge. We could always bump the revision in a later PR.

There are two changes:

Formula file saved in the bottle
Xtranslcl.c file:

diff -r 1.3.5/include/X11/Xtrans/Xtranslcl.c 1.3.5_with_fix/include/X11/Xtrans/Xtranslcl.c
84c84
< # include <sys/stropts.h>
---
> # include <sys/ioctl.h>

The formula file saved in the bottle is different. That's fine.
I think the change to Xtranslcl.c should not affect the resulting binary.

The above diff is for the installed file, not the one in the build directory

Huh. Why is it installing its .c source code files? That's weird.

It is however, NOT a shared library, but code which each consumer
includes and builds it's own copy of with various #ifdef flags to make
each copy slightly different.  To support this in the modular build
system, this package simply installs the C source files into
$(prefix)/include/X11/Xtrans and installs a pkg-config file and an
autoconf m4 macro file with the flags needed to use it.

from https://gitlab.freedesktop.org/xorg/lib/libxtrans
So you're right, you will need to rebuild the bottle.

the latest build failed to produce a bottle, so I rebased this PR on top of the latest master.

Build timed out after 5h0m0s

https://circleci.com/gh/Linuxbrew/homebrew-xorg/1366
CI is trying to install and test every formula that depends on xtrans, which is timing out. =/
@maxim-belkin That hasn't been a problem before, has it?

CI is trying to install and test every formula that depends on xtrans, which is timing out. =/
@maxim-belkin That hasn't been a problem before, has it?

$ brew uses xtrans
linuxbrew/xorg/xorg

$ brew uses linuxbrew/xorg/xorg
ddd                freerdp            gromacs            libtcod            pike               vtk
diff-pdf           ghostscript        imagemagick        links              plotutils          xclip
fltk               gifsicle           imlib2             ncview             r                  xdotool
freeglut           graphicsmagick     libepoxy           openmotif          tcl-tk

$ brew uses --recursive xtrans
amtk                   gedit                  gtk+3                  logstalgia             pypy
anjuta                 geeqie                 gtk-mac-integration    mgba                   pypy3
aravis                 geocode-glib           gtk-vnc                ncview                 pytouhou
arx-libertatis         geomview               gtkmm3                 nedit                  qalculate-gtk
awf                    ghex                   gtksourceview          negfix8                r
baobab                 ghostscript            gtksourceview3         neofetch               reposurgeon
biogeme                gifify                 gtksourceview4         nestopia-ue            rlvm
blastem                gifsicle               gtksourceviewmm        nuxeo                  simple-scan
blockhash              git-annex              gtksourceviewmm3       ocrmypdf               siril
chafa                  gitg                   gtkspell3              octave                 spaceman-diff
clearlooks-phenix      gjs                    gtranslator            opencascade            supertux
clutter-gtk            gl2ps                  gucharmap              openclonk              sxiv
corebird               glade                  gupnp-tools            opencsg                tcl-tk
ddd                    glew                   gv                     openmotif              tepl
diff-pdf               gmsh                   gwyddion               openmsx                tiger-vnc
djvu2pdf               gnome-autoar           homebank               osm-gps-map            ttygif
dmtx-utils             gnome-builder          imagemagick            pcl                    vcs
dynare                 gnome-latex            imlib2                 pdal                   vips
easy-tag               gnome-recipes          klavaro                pdf-redact-tools       visp
efl                    gnumeric               lablgtk                pdf2image              vte3
epstool                gobby                  latex2html             pdfpc                  vtk
ettercap               goffice                libchamplain           pdfsandwich            widelands
evince                 goocanvas              libdazzle              pike                   xclip
fceux                  gource                 libepoxy               pioneers               xdot
feh                    gpredict               libgweather            plotutils              xdotool
fig2dev                grace                  libgxps                plowshare              yelp-tools
file-roller            gradio                 libinfinity            ppsspp                 zbar
fltk                   graph-tool             libpeas                pqiv                   zenity
freeglut               graphicsmagick         libspectre             prefixsuffix           zim
freerdp                gromacs                libtcod                ps2eps
fs-uae                 gsmartcontrol          links                  pstoedit
gdl                    gspell                 linuxbrew/xorg/xorg    pygtksourceview

IMHO, the easiest solution is to not use linuxbrew/xorg/xorg as a dependency. 🙄

My best suggestion for now is to build this bottle locally (within Docker).
Another possible option would be to add an environment variable to brew test-bot to disable testing recursive dependents, and test only immediate dependents. Then that env var could be enabled for a single PR by some hackery (adding it to .circleci/config.yml for a single PR, for example).

yeah... I'll remove the bottle block here then and build a bottle manually


#415 generate instance after validation

I moved the instance creation after the validation,
and fixed issues with none UTC time zones and issues with cultures than don't behave like the invariant culture.
As it was the only way I could get all the unit tests to pass.
So the pull request got a bit bigger than expected

I seperated the different parts into three commits so that it should be possible to review the code changes via the individual commits to get around the massive snapshot commit

Looks good to me - thank you so much for the contribution!

@ThomasSkyldahl - if you get a chance, would you be able to see if this branch builds on your setup? I rebased my branch onto main once I had merged your branch to main. I think everything is OK and everything builds for me locally, but it would be great to confirm that it builds for you too given that you experienced issues previously.
Thanks in advance!
https://github.com/SteveDunn/Vogen/pull/437

@SteveDunn I will try it out tonight 👍

@SteveDunn everything compiles and all the tests pass at my end!

Excellent - thank you @ThomasSkyldahl !


[msal-node] Use CryptoProvider to generate PKCE codes in Auth Code PKCE sample application

This PR adds the use of the CryptoProvider class to actually generate the PKCE codes in the auth-code-pkce AAD standalone sample and the EndToEndTestApp Auth Code scenario.

Out of curiousity, why do we make applications perform these steps themselves, instead of abstracting it away like we do for MSAL Browser?

@hectormmg can we do the same for b2c-auth-code-pkce sample?

Out of curiousity, why do we make applications perform these steps themselves, instead of abstracting it away like we do for MSAL Browser?

@jasonnutter I may be wrong here but I believe it was left out of the scope since msal-node would mostly be used for confidential clients. We've talked about abstracting it away eventually but it's not currently a priority.

@hectormmg can we do the same for b2c-auth-code-pkce sample?

@derisen yes, I'll try to make it part of this PR if it doesn't get too large!


WIP Default iterations per namespace for cni heavy

Description
Fixes ...
... failures running node-density-heavy and node-density-cni @245ppn on clusters with more than 24 worker nodes.

Thoughts on renaming the workload to cni-heavy-v2? I fear people might just compare the results when they shouldn't really be compared...

My thoughts on that are: this is intended to be a replacement. I don't think the current configuration (245ppn in 1 namespace) should be run at all going forward.
Maybe we have a cutover period in the next readouts that run old & new config, but honestly the current config doesn't provide meaningful results, hence trying to erase it.

My thoughts on that are: this is intended to be a replacement. I don't think the current configuration (245ppn in 1 namespace) should be run at all going forward.

ack. ➕  My concern is people (not in our team) taking this new test and comparing it to prior runs.

Maybe we have a cutover period in the next readouts that run old & new config, but honestly the current config doesn't provide meaningful results, hence trying to erase it.

I consulted the two known consumers (QE and ACS), and neither are using ocp wrapper to run cni and heavy that they would be affected by this change.
The e2e-benchmarking parameters from kube-burner-ocp-wrapper use the defaults defined in the wrapper. I learned from Paige they have not migrated the jobs with cni or heavy to the e2e ocp-wrapper, so I proposed  to wait until these changes are merged and can work at that time to (hopefully) get more meaningful results out of the test.
I added missing documentation for the config variable, and a note to clarify the behavior.
If we want to add some note to the next release notes or something to call it out, I am happy to do that.

Looks like a flake to me. I can't find a re-run option anywhere.


Is there any example that how to use replaceable components?

import { ToastProvider } from 'react-toast-notifications';

const MyCustomToast = ({ appearance, children }) => (
  <div style={{ background: appearance === 'error' ? 'red' : 'green' }}>
    {children}
  </div>
);

const App = () => (
  <ToastProvider components={{ Toast: MyCustomToast }}>...</ToastProvider>
);

You could take a look at the snack implementation from the example website, and where it's passed to the provider.

You could take a look at the snack implementation from the example website, and where it's passed to the provider.


AbstractPlotting.set_theme! does not work for some attributes

A simple example:
using Makie

set_theme!(Attributes(color = :red, linewidth = 3))
p = lines([1, 2], [1, 2])

Despite this, p shows a black line with the default linewidth.
Some other attributes work, e.g.
set_theme!(Attributes(axis2d = Attributes(grid = Attributes(linewidth = (10, 10)))))
p = lines([1, 2], [1, 2])

correctly shows thick gridlines.
This issue is observed with both GLMakie and CairoMakie.
Versioninfo:
julia> versioninfo()
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
  JULIA_CUDA_USE_BINARYBUILDER = false
  JULIA_NUM_THREADS = 2

Project info:
julia> Pkg.status()
Project themetest v0.1.0
Status `C:\Users\invis\coding\julia\dev\themetest\Project.toml`
  [537997a7] AbstractPlotting v0.11.2
  [13f3f980] CairoMakie v0.2.8
  [5ae59095] Colors v0.12.2
  [ee78f7c6] Makie v0.11.0

This same issue is still happening, though I found it using with_theme. Font size works fine, various tick attributes under Axis work fine, but linewidth does not.
theme = Theme(linewidth=5)
with_theme(theme) do
    lines([1,2], [1,2])
end # produces thin line
lines([1,2], [1,2], linewidth=5) # produces thick line


Purchase order to bill conversion

[x] Vendor switch can't possible at the time of conversion. (same like sales order conversion)
[x] Vendor can't change after conversion from purchase order (bills edit).

issue fixed !


Validate Backup

Hi @illusional, thanks so much for the review! That was super helpful in cleaning up + securing this script. Hopefully, I've addressed everything now -- let me know if not.

Hey @illusional thanks so much for the review! I've tried to address all of the feedback provided. I seem to have forgotten what you said would be involved for this step though (In terms of testing the SM API)

Update the configuration to point to the new VM as the production VM.

https://github.com/populationgenomics/sample-metadata/blob/disaster-recovery/db/backup/README.md#yearly-procedures


Galley segfault under kubernetes integration2 testing framework

When I tried to use new testing framework for my test, I ran cmd as below
 go test -v ./tests/integration2/security/healthcheck  -istio.test.noCleanup   -istio.test.env  kubernetes \
-istio.test.kube.helm.values "global.hub=gcr.io/jianfeih-test,global.tag=0302a"  \
--istio.test.kube.suiteNamespace  istio-suite   --istio.test.kube.testNamespace istio-test

Istio system can not be marked as ready, and Pilot is haning on Galley's readiness. While I found errors about mcp connection
2019-03-05T19:04:23.127428Z     info    Handling event update for pod istio-galley-74b55bf8df-9gwqw in namespace istio-system -> <IP_ADDRESS>
2019-03-05T19:04:23.286403Z     info    mcp     (re)trying to establish new MCP stream
2019-03-05T19:04:23.286755Z     error   mcp     Failed to create a new MCP stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:9901: connect: connection refused"

I looked at Galley side, it seems  that has been restarted several times. Look at logs, I found some null pointer dereference.
2019-03-05T19:01:33.045645Z     info    Istio Galley<EMAIL_ADDRESS>Starting gRPC server on tcp://<IP_ADDRESS>:9901
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1a798b2]

goroutine 34 [running]:
istio.io/istio/galley/pkg/server.(*Server).Run(0x0)
        /workspace/go/src/istio.io/istio/galley/pkg/server/server.go:217 +0x22
istio.io/istio/galley/pkg/server.RunServer(0xc4200decc0, 0x21fad00, 0xc42001ee10, 0x21fad00, 0xc42001ee60)
        /workspace/go/src/istio.io/istio/galley/pkg/server/server.go:277 +0x1b3
created by istio.io/istio/galley/cmd/galley/cmd.GetRootCmd.func2
        /workspace/go/src/istio.io/istio/galley/cmd/galley/cmd/root.go:88 +0x3c8

What Istio version are you using? This should be fixed now https://github.com/istio/istio/issues/11748

Figured it out, due to image was stale. Closing.


Cleans up Jenkinsfile, and Makes It Less Brittle

Also, we need to address how Jenkins is displaying our stages.  Its collapsing all of xenial in the Blue Ocean View, and also errors are not being passed up.  Builds are failing, being reported to the Jenkins/Slack bot, but the build continues, and hides where the error occurred.


SW-6611 Caseload list page for the New Deputy Orders team

Codecov Report
Patch coverage: 25.64% and project coverage change: -1.66 :warning:

Comparison is base (f293009) 65.78% compared to head (7ffbfe8) 64.13%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #345      +/-   ##
==========================================
- Coverage   65.78%   64.13%   -1.66%
==========================================
  Files          22       23       +1
  Lines        1102     1132      +30
==========================================
+ Hits          725      726       +1
- Misses        329      357      +28
- Partials       48       49       +1

Impacted Files
Coverage Δ

internal/model/client.go
59.25% <0.00%> (-29.63%)
:arrow_down:

internal/model/date.go
0.00% <0.00%> (ø)

internal/sirius/get_client_list.go
44.11% <50.00%> (ø)

internal/model/team.go
100.00% <100.00%> (ø)

internal/server/caseload.go
86.95% <100.00%> (-1.05%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Create a new date widget with dropdowns (SDC)

Problem:
The current date picker has poor usability, especially for dates far in the past like date of birth. The main challenge is selecting a date and month using the calendar entry method. Typing the date with the keyboard is challenging due to missing / and adhering to the date format.
Proposed solution:

Create a new date widget using dropdowns for day, month and year.
Add it to the SDC catalog app as Date of Birth widget (open to other ideas)
Keep existing date picker as is.

Alternatives:
Change the existing date picker (we are proposing minor improvements in #1802)
Use a different or legacy date picker (ice box issue - will create issue)
Design mocks:

@shelaghm Will we make this widget configurable such that you can define with spinner comes first?
E.g if the expected date format is yyyy/mm/dd then the year spinner is the first one

For the existing date picker, having the / automatically added at the least would still be good to do because if you don't have that the entry seems almost unusable for these users. So it'd be nice for that to be part of the proposed solution. I think this date entry would be the most familiar one for new users.
For the second date option using drop downs, I like it though my two concerns are (1) the tap area is small (for this user) in this style drop down, and (2) the day range changes based on the month (I guess legacy spinners displaying the three columns were a way around this). But this is nice and is great for when there's a default date already entered (like today's date).

@dubdabasoduba Yes, exactly as you said. It will need to be configurable so that the developer can define their preferred date format and the dropdown sequence would shift accordingly.
@rowo Helpful to hear your feedback. Our plan is to do both, (1) add auto slashes in #1802 and (2) add this new component. It will then be very helpful to hear from you which one you end up using in your app and when/why.
Very valid concerns, thanks for raising those.
(1) Yes, would be ideal to have larger tap area. Alternative could be to stack vertically.
(2) An alternative could be to do keyboard entry for day or year instead of a dropdown. Or validate the date after month has been entered, and an error message is then shown if the date is not a valid date.
Feel free to suggest other ideas if something comes to mind. Thanks!

@rowo How did you address this in OpenSRP?


Improve sitemap search speed

heuristics to reduce the search space
better server to improve speed
right now it's just on the edge of painfully slow

redis, load the whole possible sites table into memory

Custom rewrite in C/Rust/Go for this one feature? I'm sure that psql is well made, but given the simplicity of this query (fuzzy search on 2 fields) it might be more efficient to do outside of SQL queries. The actual logic is pretty straightforward and can be done in a single pass with no backtracking.
With that said its probably possible to write the query better and have similar advantages. Things GROUP BY are probably not necessary.

Also I'm bad at C/Go and do not actually know rust

https://www.charityapi.org/post/improving-postgres-text-search-speed-7x-on-millions-of-records

Can strip out spaces, another column of the cleanest version. Might improve query speed

hold up
https://api.ultimate-guitar.com/api/v1/tab/suggestion?q=peach

replaced sitemap search with api search


Support for Hugging Face Local Models

One question regarding support for Hugging Face local model, I found this:
https://github.com/stanford-crfm/helm/blob/7ead1d6e41e91683ff6b06a94bd9e2ed7b629d0a/docs/huggingface_models.md?plain=1#L13
saying "Models at local file paths are not supported."
Also, I have found this link: https://github.com/stanford-crfm/helm/pull/1505, which I am not able to find mentioned files. Could you please confirm the first link is the most updated and accurate one?
Many thanks!

The docs are outdated - we do support Hugging Face models at local file paths now.

Hi @NoushNabi, I have added documentation for using Hugging Face models at local paths at the Adding New Models and Hugging Face Model Hub Integration pages.


AttributeError when serializing int values

After loading in a properties file, if I use __setitem__ to set a value to a non-string (e.g. prop['http.port'] = 7180), I get an attribute error upon trying to serialize my Properties instance with str(). That is,
str(props)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/site-packages/jproperties.py", line 58, in __str__
    ret.append(self.escape_key(node.key) + node.separator + self.escape(node.value))
  File "/usr/lib/python3.5/site-packages/jproperties.py", line 81, in escape
    return value.encode("unicode_escape").decode("utf-8")
AttributeError: 'int' object has no attribute 'encode'

Perhaps a forced cast should be introduced somewhere in the escape chain?

Yeah, probably a good idea to str() stuff when you set it.

Can the module be updated to put values through str() automatically when serializing?


FluidDataStoreRuntime: mixins or hooks?

This is follow up to https://github.com/microsoft/FluidFramework/pull/3822
We need to figure out if we want to continue strategy of mixins (requestFluidDataStoreMixin) vs. strategy of hooks (registerRequestHandler).
I favor mixins - we need to fix all examples to use mixins to prove that's reasonable strategy.

Please see following PRs:
https://github.com/microsoft/FluidFramework/pull/4216
https://github.com/microsoft/FluidFramework/pull/4215

PRs mentioned above are checked in, we are settling for mixins.


Support secondary y-axis

Greeting,
Is there a plan to add secondary y-axis support? It seems not supported by v0.0.6 and Grafana V6.4.4.
Best,
bjwxh

+1 for a secondary y-axis.

+1 for secondary y-axis. Trying to plot voltage and current magnitudes in same plot. Thanks!


Missing wireguard dependency

I was following the instructions for installing tailscale here: https://tailscale.com/kb/1036/install-arch. Watching journctl logs I saw the errors:
Feb 17 11:42:37 kabuto relaynode[31062]: 2020/02/17 11:42:37 CreateTUN: no such device
Feb 17 11:42:37 kabuto relaynode[31062]: 2020/02/17 11:42:37 Error starting wireguard engine: no such device
Feb 17 11:42:37 kabuto systemd[1]: tailscale-relay.service: Main process exited, code=exited, status=1/FAILURE

Then I realized that I had didn't have wireguard installed. So I installed wireguard-arch and wireguard-tools, and it works now. Should we add those as dependencies, or is it intentionally left out because there are several ways to install it

Tailscale should not depend on the kernel wireguard implementation, currently we only use the userspace Go implementation.
Based on the error, I suspect you upgraded your kernel, and so the tuntap module failed to load (because it was for a newer kernel version). Can you try removing the wireguard packages and rebooting to resync your kernel with your installed modules?

Oh interesting. Yes removing those packages, then rebooting, shows no journalctl errors. Thanks


Remove nul.ls

It triggered a bug (see below).
According to Wikipedia, it can only be used for sub-registration by the National University of Lesotho.

This is what happened in github.com/domainr/whois when I bumped the ZoneDB dep there:
$ go get -u github.com/zonedb/zonedb
go: github.com/zonedb/zonedb upgrade => v1.0.2890
go get<EMAIL_ADDRESS>verifying module<EMAIL_ADDRESS>reading<EMAIL_ADDRESS>410 Gone
	server response: not found: create zip: malformed file path "metadata/nul.ls.json": "nul" disallowed as path element component on Windows

Removing this zone is the quick fix, unfortunately.

Wow.


python command for OS X

Hello!
Please review initial implementation of "python" command for now only for OS X.
You can install any package from https://www.python.org/downloads/mac-osx/ using /usr/sbin/installer and uninstall it correctly using pkgutil
Initial test for 2.x and 3.x is here, you can make branch if it's ok, and I'll test it thoroughly and implement todos, or let me know if it should be refactored.
10723 ◯  hatch python 3.6.0
Downloading python 3.6.0 from python.org...
Saved installer in /Users/a1/Downloads/python-3.6.0-macosx10.6.pkg
Installing interpreter (may require your sudo password)...
Interpreter installed in /Library/Frameworks/Python.framework
MacBooks-MacBook-Pro-2 ॐ  ~:
10724 ◯  python3.6 -c "import sys; print(sys.executable)"
/usr/local/bin/python3.6
MacBooks-MacBook-Pro-2 ॐ  ~:
10725 ◯  python3.6
Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
MacBooks-MacBook-Pro-2 ॐ  ~:
10726 ◯  hatch python 3.6.0 --rm
Uninstalling python in /Library/Frameworks/Python.framework/3.6
MacBooks-MacBook-Pro-2 ॐ  ~:
10727 ◯  python3.6
zsh: command not found: python3.6
MacBooks-MacBook-Pro-2 ॐ  ~:
10728 ◯

Hey, now coverage is ok, have some ideas for refactoring? I think I can do install on windows in same manner and refactor.

@lulunevermind I'll check this out soon! I have an idea for Windows; actually a cross-platform solution.


create DNS record for pinot servers before the pods are ready

During pinot-server startup, the segments are marked online but brokers are not able to access them as the DNS record for the pod will not be created till the pod is in ready state. Hence, the queries fails during the startup period.
Now, the pod DNS record will be created even if the pod is not ready.

During pinot-server startup, the segments are marked online but brokers are not able to access them as the DNS record for the pod will not be created till the pod is in ready state. Hence, the queries fails during the startup period.
Now, the pod DNS record will be created even if the pod is not ready.

If the pod isn't marked as ready but the broker queries are able to access them, will that cause any issues? What is still unhealthy about the server such that it is not yet marked ready?

During pinot-server startup, the segments are marked online but brokers are not able to access them as the DNS record for the pod will not be created till the pod is in ready state. Hence, the queries fails during the startup period.
Now, the pod DNS record will be created even if the pod is not ready.

If the pod isn't marked as ready but the broker queries are able to access them, will that cause any issues? What is still unhealthy about the server such that it is not yet marked ready?

Server is waiting for all the segments to be loaded before pod is marked as ready. In the meanwhile, broker can route queries for the segments which are already loaded and marked as online in zookeeper metadata.

During pinot-server startup, the segments are marked online but brokers are not able to access them as the DNS record for the pod will not be created till the pod is in ready state. Hence, the queries fails during the startup period.
Now, the pod DNS record will be created even if the pod is not ready.

If the pod isn't marked as ready but the broker queries are able to access them, will that cause any issues? What is still unhealthy about the server such that it is not yet marked ready?

Server is waiting for all the segments to be loaded before pod is marked as ready. In the meanwhile, broker can route queries for the segments which are already loaded and marked as online in zookeeper metadata.

I don't know enough about how pinot works, so just want to clarify - if I do something like a COUNT(*) and some segments are online and some aren't, will it return based on the loaded segments only? In other words, are we trading errors for potentially incorrect results? If pinot knows enough to not return queries unless it has all relevant segments available, then this change make sense. If it will start returning inaccurate results, I'd rather see errors.

When pinot server is shut down, all the segment replicas hosted by it are marked as OFFLINE.

During server startup, it loads segments one by one and mark them as ONLINE in the metadata. This process can take 3-5 minutes. Once all the segments are loaded, the server pod transitions to ready state and headless service creates a DNS record for the pod.
If a query is executed during this startup period and the segment replica is ONLINE in metadata (segment has already been loaded by the server), broker can route that query to the server pod and it fails with the following exception:
2020/10/22 08:43:34.824 INFO [CallbackHandler] [CallbackHandler-AsycSubscribe-Singleton] Subscribing to path:/traceable-views/EXTERNALVIEW took:189
2020/10/22 08:43:34.831 ERROR [QueryRouter] [jersey-server-managed-async-executor-84] Caught exception while sending request 1515 to server: pinot-server-3_R, marking query failed
java.net.UnknownHostException: pinot-server-3.pinot-server.traceable.svc.cluster.local
        at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1505) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1364) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1298) ~[?:?]
        at java.net.InetAddress.getByName(InetAddress.java:1248) ~[?:?]
        at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:202) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:48) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:182) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:168) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:985) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:505) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:416) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:475) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2020/10/22 08:43:34.833 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-84] requestId=1515,table=domainEventView_REALTIME,timeMs=7,docs=0/0,entries=0/0,segments(queried/processed/matched/consuming/unavailable):0/0/0/0/0,consumingFreshnessTimeMs=0,servers=0/4,groupLimitReached=false,exceptions=1,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-server-0_R=0,-1,0,0;pinot-server-2_R=0,-1,0,0;pinot-server-3_R=-1,-1,0,0;pinot-server-1_R=-1,-1,0,0,query=Select DISTINCTCOUNT(api_id) FROM domainEventView WHERE customer_id = '3e761879-c77b-4d8f-a075-62ff28e8fa8a' AND ( ( event_time_millis >= '1603352612948' AND event_time_millis < '1603356212948' ) AND ( ( category = 'SECURITY' AND security_event_category IN ('MALICIOUS') AND api_is_external = 'true' ) AND ( api_id != 'null' ) ) ) limit 1

Broker routes the queries to the server directly (without the k8s service). With this fix, the DNS record will be created at the start of pod and broker will be able to queries the segments which are already loaded.
The segments which are yet to be loaded are marked as OFFLINE in the metadata and broker will use any other ONLINE replica (if available) for those segments.
cc: @aaron-steinfeld @buchi-busireddy @surajpuvvada @laxmanchekka

Thanks for the detailed explanation @ravisingal
@laxmanchekka @surajpuvvada we should prioritize tuning Pinot so that reloading segments shouldn't take 3-5mins


[SMALLFIX] Remove empty java.security.krb5.realm/kdc options

After #3095, empty java.security.krb5.realm/kdc options will be added to ALLUXIO_JAVA_OPTS in all os platforms, which conflicts with the option: java.security.krb5.conf in secure hadoop env.
A simple solution is to remove the empty options and let user export krb5 options if needed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/11093/
Test PASSed.

Merged build finished. Test PASSed.

LGTM

LGTM, thanks for the fix.


It would be really nice if the pins saved so they appeared after phone restart.

I would suggest adding a feature that allows the pins to be saved. I have made pins in the past, but they don't appear after I restart my phone. I know this would be a lot of work, but for me this is the difference in a good app and an amazing app that I use everyday.

Shouldn't be that much work. I have the following idea:

Save pins state (Title, content, visibility & priority) after creation to SharedPreferences using a JSON-Array
Make use of notification-DeleteIntents to remove pins from the array
Use a BroadcastReceiver to receive BOOT_COMPLETED intents and restore pins by parsing the JSON-Array

I'll try to get it this weekend running.

Finished in version v1.5.


pull request for e2e test at 12/22/2016 21:26:18

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Update ktlint to v0.48.0

There are three new rules that I haven't added wrappers for, since we seem to have the same ones in detekt already:

Class naming
Function naming
Property naming

Also had to disable NoSemicolons rule on detekt's codebase because of https://github.com/pinterest/ktlint/issues/1733 but I'm sure that will be fixed soon. When this is merged I'll open a tracking issue for that.

There are three new rules that I haven't added wrappers for, since we seem to have the same ones in detekt already:
Class naming
Function naming
Property naming

I believe Detekt implementation for these rules is more flexible (Since ktlint is mostly on/off with limited configurability), hence I believe we should still add a wrapper rule for them (There can be cases where Detekt users turn off the detekt implementation and use ktlint). This can be a discussion / follow-up


Suddenly cant import created elu files (created with this addon)

reinstalled the plugin, reinstalled blender same issue.
worked for weeks but now even directly exporting an elu from the original blender project files, then trying to import that fails with this error.

in case it matters- im running blender 4.2.1 on windows 11.

Thanks for the bug report. I wasn't able to replicate this on my end.
Please add me on Discord and send me the .elu you're having trouble with: Krunk#6051

Solved as part of v0.9.4


fix: correct error handling and remove unused code in SendFundsForm component

Problem solved
Short description of the bug fixed or feature added
Changes made

[ ] Public API changes: list the public API changes made if any
[ ] Internal API changes: explain the internal logic changes

How to test

[ ] Automated tests: link to unit test file
[ ] Manual tests: step by step instructions on how to test

Contributor NFT
Paste in your wallet address below and we will airdrop you a special NFT when your pull request is merged.
Address:

PR-Codex overview
The focus of this PR is to refactor the error handling logic in the SendFundsForm component of SendFunds.tsx.
Detailed summary

Removed showInvalidAddressError state variable
Refactored error message handling in getErrorMessage function
Updated rendering of invalid address error message in the form

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

#3593  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @joaquim-verges and the rest of your teammates on  Graphite


Add Controller

⚒ WIP
⚠ The Playground Example has been re-built using this added router, check it out!

Description
This PR adds a page router, page routes and page controllers.
It's API allows for designing reusable, composable pages.
All routing is handled automatically. You can also push and pop results from individual pages.
This all uses showWithDynamicPath under the hood. No changes were made to existing code
This is still needs tests and docs.
I'm open to any changes on the API design.
Router
I've added a router that handles all navigation:
// Router with initial routes
final router = WoltModalSheetPageRouter([RootPage()]);
// Push it
WoltModalSheet.showWithRouter(
  context: context,
  router: router,
);

You can also go directly into child pages by using multiple routes:
// Router that will jump into the child
final router = WoltModalSheetPageRouter([RootPage(),ChildPage()]);

Page Route
When using this router, you have to implement the WoltModalSheetPageRoute class
Under the hood this will wrap you page with everything you need to create an easy to manage navigational stack.
Example Page:
class HomePage extends WoltModalSheetPageRoute {
  HomePage();

  @override
  MainContentBuilder mainContent() {
return MainContentBuilder.scrollable(
    (context, pageController) => const Column(
          crossAxisAlignment: CrossAxisAlignment.start,
          children: [
            Padding(
              padding: EdgeInsets.all(16.0),
              child: Text(
                  'Scroll to see the custom top bar with a search field in it.'),
            ),
            Placeholder(fallbackHeight: 500, color: WoltColors.blue),
          ],
        ));
  }

  @override
  Widget? stickyActionBar(context, pageController) {
    return Padding(
      padding: const EdgeInsets.all(16.0),
      child: WoltElevatedButton(
        onPressed: () async {
          // Await results from child page
          final confirmed = await pageController.sheetRouter.push<bool>(NextPage());

          // Pop to parent page
          pageController.sheetRouter.pop(confirmed);

        },
        colorName: WoltColorName.blue,
        child: Text("Next"),
      ),
    );
  }

  @override
  Widget? pageTitle(context, pageController) => ModalSheetTitle("Home");

  @override
  Widget? trailingNavBarWidget(context, pageController) =>
      const WoltModalSheetCloseButton();

  @override
  Widget? leadingNavBarWidget(context, pageController) =>
      WoltModalSheetBackButton(
          onBackPressed: () => pageController.sheetRouter.pop());

  @override
  Widget? heroImage(context, pageController) => null;

  @override
  Widget? topBar(context, pageController) => null;

  @override
  Widget? topBarTitle(context, pageController) => null;
}

Compose !
The benefit of using classes, is now we can inherit widgets and add parameters!
class ProductPage extends HomePage {
// Add parameters to make reusable
ProductPage (this.productName);

// Only change the widgets you want
//...
}

Modifable Route
This also takes care of changing route settings after it has been pushed.
The following would change a pages drag setting using a switch
ValueListenableBuilder(
  valueListenable: pageController.properties,
  builder: (context, value, child) {
    return Switch(
      value:
          pageController.properties.value.enableDrag ?? false,
      onChanged: (newValue) {
        pageController.properties.value = pageController
            .properties.value
            .copyWith(enableDrag: newValue);
      },
    );
  },
)

I would love to get feedback on this!
Related Issues
Replace this paragraph with a list of issues related to this PR from the issue database. Indicate, which of these issues are resolved or fixed by this PR.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process.

[ ] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] My PR includes tests for all changed/updated/fixed behaviors.
[ ] All existing and new tests are passing.
[ ] I updated/added relevant documentation (doc comments with ///).
[ ] The analyzer (melos run analyze) does not report any problems on my PR.
[ ] The package compiles with the minimum Flutter version stated in the pubspec.yaml

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change.
[x] No, this is not a breaking change.

This PR is still a WIP.
I've used this implementation on an app I'm building and I'm finding lots of issues with it.
Once I've got time, I will modify it do fix a whole slew of issues.
As far as the package, thanks so much , it's awesome!
I wanna make it easier for everyone to use!


Added another installation method

Added NIx/NixOS installation method

@BurntSushi fixed ;)

I think the text is still wrong. It says:
If you're a Nix/NixOS user, you can install RIPGREP from nixpkgs

@michelboaventura Thanks for the heads up. I fixed it.


fix(ssa): Handle array arguments to side effectual constrain statements

…ffects
Description
Problem*
Resolves the error that occurs in this PR (https://github.com/noir-lang/noir/pull/3739)
No issue as was discovered and fixed quickly.
Summary*
Using an array as an argument in a side effectual constrain statement will cause this panic:
Message:  internal error: entered unreachable code: Can only cast to a numeric
Location: compiler/noirc_evaluator/src/ssa/acir_gen/mod.rs:1652

Here is an example program which would previously fail that is included as a regression test:
struct Bar {
    inner: [u32; 3],
}

fn main(y: pub u32) {
    let bar = Bar { inner: [100, 101, 102] };

    // The assert inside the if should be hit
    if y < 10 {
        assert(bar.inner ==  [100, 101, 102]);
    }

    // The assert inside the if should not be hit
    if y > 10 {
        assert(bar.inner == [0, 1, 2]);
    }
}

Without this fix it is not possible to constrain arrays inside of if statements based upon witness conditions.
Additional Context
Documentation*
Check one:

[X] No documentation needed.
[ ] Documentation included in this PR.
[ ] [Exceptional Case] Documentation to be submitted in a separate PR.

PR Checklist*

[X] I have tested the changes locally.
[X] I have formatted the changes with Prettier and/or cargo fmt on default settings.

Would we still need this fix if assertions just took a boolean in SSA and the equality was a separate instruction?

Would we still need this fix if assertions just took a boolean in SSA and the equality was a separate instruction?

I don't think so, as we would only be constraining on the boolean. I went with this solution for now as this seems like an easy bug to run into and making equality a separate instruction is a larger change.


Receiving the following error

My knowledge of coding is nil. But I can follow instructions. It should be working as far as I can tell. Please help!
Thanks in advance!

Oh, sorry. important bit of information I am using the old leaked SD model and assumed if the final names were the same there wouldn't be an issue. This obviously, could be the issue.

Is it possible you’re using the config file for one model and are trying to load the weights for another? If you are using the leaked weights then be sure to use the stable-diffusion-v1 YAML file.

Hey, I'm getting EXACTLY the same result... I'm not sure I follow what you're saying the solution is.  I'm using the model.ckpt linked   in the readme.md here.  Mainly I'm just trying to get prepared so that I can drop in the proper model that hopefully drops tomorrow.

Thanks for picking that up! I have fixed the README.


Quick Fix for variable name propagation issue in #31

Description
Change of variable name in here did not go throughout the function and we are using the old name
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

This type of simple issue is extremely unnoticeable & easily sneak in through the person who develops but almost always gets caught by the reviewer. Let's implement no-merge-without-one-approval policy, perhaps?

Good catch!
I think we're probably at the point where we should do mandatory reviews. Why don't we set the 0.1.0 release as the official start of that policy?


python310Packages.greeclimate: 1.3.0 -> 1.4.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.greeclimate is: "Discover, connect and control Gree based minisplit systems"
meta.homepage for python310Packages.greeclimate is: "https://github.com/cmroche/greeclimate"
meta.changelog for python310Packages.greeclimate is: "https://github.com/cmroche/greeclimate/blob/refs/tags/v1.4.0/CHANGELOG.md"
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.4.0 with grep in /nix/store/g2ysy2r7kmk1mmy8vzw4vlb652l690ch-python3.10-greeclimate-1.4.0
found 1.4.0 in filename of file in /nix/store/g2ysy2r7kmk1mmy8vzw4vlb652l690ch-python3.10-greeclimate-1.4.0

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath
home-assistant-component-tests.gree
python310Packages.greeclimate
python39Packages.greeclimate

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/g2ysy2r7kmk1mmy8vzw4vlb652l690ch-python3.10-greeclimate-1.4.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.greeclimate https://github.com/r-ryantm/nixpkgs/archive/1130d99e67ee95b6b2354473bf52cc6d85d0d9fb.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/g2ysy2r7kmk1mmy8vzw4vlb652l690ch-python3.10-greeclimate-1.4.0
ls -la /nix/store/g2ysy2r7kmk1mmy8vzw4vlb652l690ch-python3.10-greeclimate-1.4.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

home-assistant-component-tests.gree
python310Packages.greeclimate
python39Packages.greeclimate

Maintainer pings
cc @dotlambda for testing.

Result of nixpkgs-review pr 204961 run on x86_64-linux 1

3 packages built:

home-assistant-component-tests.gree
python310Packages.greeclimate
python39Packages.greeclimate


Implement HttpServletRequest.getRequestURL()

Implementing getRequestURL() on the HttpServletRequest object addresses https://github.com/awslabs/aws-serverless-java-container/issues/10.  In the process, I also adjusted the following methods, which all contribute to getRequestURL()

getServerName()
getServer/LocalPort()
getContextPath()

Tests added and AwsProxyRequestBuilder adjusted

Thanks Matt, merged the pull request


[patch] Fix err variable using parent scope because of mispelled parameter

Hello!
A coworker was getting some weird errors when he tried to lift a second app on a port that was already in use. It was odd to me that this error didn't propagate to the err parameter in the sails lift callback. After checking out the source, it appears that there was a typo in the final handler of sails.runBootstrap. Instead of receiving the err value from the async call on line 82, the err variable is instead referencing the one defined by runBootstrap on line 68.
I hope this helps!

@dustinws very good catch.  Thanks so much!

Glad to help, thanks!