KMeans output is empty

Hi
I have tried to generate date by using "Data Generator - KMeans"  to this path (/tmp/kmeans-data-generation.csv) but it give output of folder that contain an empty ".csv" file that name "_SUCCESS"  with the following waring :

This generated file does not contain any data.
But if i chose the output of KMeans data generator to "console" it show the output correctly
I am not sure what is the issue,, I appreciate any help

Hi @Aalnafessah! Glad you're using Spark-Bench :)
The output functions of Spark-Bench are just wrappers around Dataframe.write(). See method here: https://github.com/SparkTC/spark-bench/blob/master/utils/src/main/scala/com/ibm/sparktc/sparkbench/utils/SparkFuncs.scala#L25 So my guess would be this is quirk of your Spark setup and not an intrinsic issue with Spark-Bench.
One of my colleagues at work has noticed that on a particular cluster he has to use some variation for local files. It’s either “local:///tmp/KMeen.csv” or “file:///tmp/KMeen.csv”, can’t remember which, but he had a similar issue where just "/tmp/KMeen.csv” wasn’t working.
A good way to test this would be to open up a spark-shell totally outside of Spark-Bench but on the same system, and just how the output works if you write a super simple dataframe. Like:
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.

scala> val data = Array(1, 2, 3, 4, 5)
data: Array[Int] = Array(1, 2, 3, 4, 5)

scala> val distData = sc.parallelize(data)
distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:26

scala> import spark.implicits._
import spark.implicits._

scala> val df = distData.toDF
df: org.apache.spark.sql.DataFrame = [value: int]

scala> val output = "/tmp/test.csv"
output: String = /tmp/test.csv

scala> df.write.option("header", "true").csv(output)

Thanks @ecurtin for you fast response.
I have Spark 2.2.0 instal as standalone cluster of three machines ( 1 master and two workers ).
I tried your example for spark-shell totally outside of Spark-Bench on same system and it worke fine. Both "/tmp/KMeen.csv” and   file:///tmp/KMeen.csv are working fine for me. But “local:///tmp/KMeen.csv” give some error. Here is the output of the example of spark-shell that shows the csv files that has been generated :

But with the output of KMeans data generator is still empty as shown here:

What i understand is that  KMeans data generator could not generate output to csv file. Is there any way to overcome this issue ?

Can you copy-paste your exact configuration file for me?

This is the conf file "KMeans-DG.conf":
spark-bench = {   spark-submit-config = [{     workload-suites = [       {         descr = "*****  KMeans data gen *********"         benchmark-output = "console"         workloads = [           {              name = "data-generation-kmeans"              rows = 10              cols = 4              output = "file:///tmp/KMeen.csv"           }         ]       }     ]   }] }
This is the command that i use:
./bin/spark-bench.sh ./examples/KMeans-DG.conf --master spark://xx.xx.xx.xx:7077
xx.xx.xx.xx is the master ip address

Hmm. Ok, so I do see a problem here, but I'm not sure if it's related to your file output. So let's address this and see what happens.
Passing --master has no effect. ./bin/spark-bench.sh only takes in one argument, which is the path to your configuration file. (I should make a ticket to make this an explicit error)
There are two ways you can set the master. I suppose there are actually three, as it might work if your spark environment configuration (outside of Spark-Bech) has master configured. Otherwise, you can also set it in an environment variable, but the better way, and the way I recommend, is to set it in your configuration file, like this:
spark-bench = {
  spark-submit-config = [{
    spark-args = {
      master = "spark://xx.xx.xx.xx:7077"
    }
    workload-suites = [
    {
      descr = "*** KMeans data gen *******"
      benchmark-output = "console"
      workloads = [
        {
          name = "data-generation-kmeans"
          rows = 10
          cols = 4
          output = "file:///tmp/KMeen.csv"
        }
      ]
    }
    ]
  }]
}

Since it sounds like you've had successful runs with Spark-Bench, master must be getting picked up either from the configuration of your Spark install or from an environment variable, because --maser is invalid.
In any case, using the configuration file you pasted (I named it aalnafessah.conf) on my local machine, (master = "local[2]") I have a successful run:
$ ./bin/spark-bench.sh examples/aalnafessah.conf
 *** SPARK-SUBMIT: [/opt/spark-2.1.1-bin-hadoop2.7/bin/spark-submit, --class, com.ibm.sparktc.sparkbench.cli.CLIKickoff, --master, local[2], /Users/ecurtin/Downloads/spark-bench_2.1.1_0.2.2-RELEASE/lib/spark-bench-2.1.1_0.2.2-RELEASE.jar, {"spark-bench":{"spark-submit-config":[{"workload-suites":[{"benchmark-output":"console","descr":"*** KMeans data gen *******","workloads":[{"cols":4,"name":"data-generation-kmeans","output":"file:///tmp/KMeen.csv","rows":10}]}]}]}}]
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/05 12:29:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/05 12:29:33 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/11/05 12:29:35 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/11/05 12:29:35 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/11/05 12:29:35 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/11/05 12:29:35 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/11/05 12:29:35 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/11/05 12:29:35 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/05 12:29:36 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/05 12:29:36 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/05 12:29:36 INFO FileOutputCommitter: Saved output of task 'attempt_20171105122936_0001_m_000001_0' to file:/tmp/KMeen.csv/_temporary/0/task_20171105122936_0001_m_000001
17/11/05 12:29:36 INFO FileOutputCommitter: Saved output of task 'attempt_20171105122936_0001_m_000000_0' to file:/tmp/KMeen.csv/_temporary/0/task_20171105122936_0001_m_000000
*** KMeans data gen *******
+------+-------------+---------+----------+---------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+--------------------+
|  name|    timestamp| generate|   convert|     save|total_runtime|scaling|numCols|              output|  k|input|numPartitions|numRows|run|spark.driver.host|spark.driver.port|hive.metastore.warehouse.dir|          spark.jars|      spark.app.name|spark.executor.id|spark.submit.deployMode|spark.master|       spark.app.id|         description|
+------+-------------+---------+----------+---------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+--------------------+
|kmeans|1509902973836|164829581|1716066737|681732843|<PHONE_NUMBER>|    0.6|      4|file:///tmp/KMeen...|  2|     |            2|     10|  0|     <IP_ADDRESS>|            62484|        file:/Users/ecurt...|file:/Users/ecurt...|com.ibm.sparktc.s...|           driver|                 client|    local[2]|local-1509902973666|*** KMeans data g...|
+------+-------------+---------+----------+---------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+--------------------+

And I get the desired output:
$ cat /tmp/KMeen.csv/*
0,1,2,3
1.8270485047568874,1.4710527183724604,-1.5197866190254334,-1.7711844582389404
0.6219314622769577,1.113822964741109,-0.9773162514847056,-0.7579784841391045
1.7465164231324293,1.4889575936956514,-0.0503504938463315,-0.303724265255192
0.5617353623246156,1.051827179649743,-1.310881195342784,0.20434322742040967
2.040964654340478,-0.27542822800237676,1.036641159426876,-2.340304518778751
0,1,2,3
1.918711918221884,0.6492353823417113,-0.9105415842564744,-1.2232318176857362
0.7341645256607737,-0.3868184653564506,0.23028151334280977,-1.9924592519362037
0.5351835184559727,-0.1919139685953814,-0.9275239890714504,0.07330992388824076
2.1483351669714503,1.278789344023917,1.8077303201534083,-0.6180897559364122
0.8745069290674965,1.2829439241238543,-1.3657981937591184,-1.4498955771429336

Here's what I suggest next for debugging:

Try re-running your config file using spark-args = { master = "spark://..."}. I doubt this is related to your issue, but it's worth a shot.
If your cluster has HDFS (Most Spark clusters do), try using "hdfs:///tmp/KMeen.csv" and see if that writes successfully.
Failing those two, my next of line of debugging inquiry would revolve around permissions issues.

I'm sorry this isn't working out of the box for you :(
Hopefully we can get to the bottom of this!

^^ For the debugging above on my local machine, I was using the distribution spark-bench_2.1.1_0.2.2-RELEASE_47.tgz downloaded from here: https://github.com/SparkTC/spark-bench/releases/tag/v47

@ecurtin
I have run the same config file using spark-args = { master = "spark://xx.xx.xx.xx:7077"}
There is still just s folder that is named (KMeen.csv) and there nothing inside this folder. Also i have run the command ./bin/spark-bench.sh examples/aalnafessah.conf as sudo  but still genreated an empty folder (KMeen.csv)
When i tried using "hdfs:///tmp/KMeen.csv"  i got :
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs:/tmp/KMeenAlnafessah2.csv, expected: file:///
Which it mean that i can use hdfs.
I think there is an issuer in the function : https://github.com/SparkTC/spark-bench/blob/master/utils/src/main/scala/com/ibm/sparktc/sparkbench/utils/SparkFuncs.scala#L37
which does not allow dataframe to be saved as csv file.
BTW, I am using Spar2.2.0 and the spar-Bench version is : spark-bench_2.1.1_0.2.2
Any suggestion ?

Hmmm, ok, that makes me wonder if this is a Spark version compatibility issue. I am using Spark 2.1.1 locally, and the current release of Spark is built against the jars for Spark 2.1.1.
I don't want to be dismissive of your intuition that there's something wrong with the function, but I have several reasons for suspecting that is not actually the case:

I am able to write CSV and Parquet from not only the KMeans data generator, but the other workloads as well.
The write functionality of the KMeans data generator is exercised in a unit test here: https://github.com/SparkTC/spark-bench/blob/v47/cli/src/test/scala/com/ibm/sparktc/sparkbench/datageneration/KMeansDataGenTest.scala#L44. Specifically, this test writes data to a temporary file on disk and then reads back that file and makes sure it has the correct number of lines here: https://github.com/SparkTC/spark-bench/blob/v47/cli/src/test/scala/com/ibm/sparktc/sparkbench/datageneration/KMeansDataGenTest.scala#L75. This, along with all the other unit tests must pass in TravisCI order for the release to be created: https://travis-ci.org/SparkTC/spark-bench/builds/295464919#L3751

So I'm very happy to be proved wrong, but those are my two reasons for not suspecting an issue with the write function itself.
I think there's a very strong possibility that this a Spark version thing. I'm going to setup a test in a branch on my fork that will run the unit tests in TravisCI against Spark 2.2.0 and that should reveal more information. Hopefully it will reproduce your bug!

@ecurtin
I will wait and see if you have the same issue with the spark version 2.2.0 or not.
Many thanks for your valuable support

Well that's annoying. It passed: https://travis-ci.org/ecurtin/spark-bench/builds/297655589 I was hoping it would fail and then the issue would be obvious. No such luck :(
I'm still not ruling out Spark version compatibility as the source of this. Unfortunately, I can't look into this anymore today, but I will dig in again tomorrow.
In the meantime, if you can let me know what Hadoop version you're running on your cluster (probably 2.6, 2.7, or 2.8) that might be a clue. If you have the time to try to install Spark 2.1.1 yourself and rerun your Spark-Bench job, that would be helpful, but I know that's asking a lot :)
More tomorrow...

I using spark-2.2.0-bin-hadoop2.7
I will work on this issue until midnight and i will see if the issue can be solved ,otherwise i will wait for you tomorrow before tring to install 2.1.1     : )

Nothing conclusive to update you with yet, but haven't forgotten about you!

@ecurtin
Thanks for you valuable consideration.
With Spark 2.2.0 , saving the dataframe to csv file is genrating a folder with multiple csv files inside this folder. Does this affect the generation of csv file in Sparkbench ?

@ecurtin
Thanks for you valuable consideration.
In your logs you have this line :
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

But in my logs i have these lines, Which i belive they reflect the issues:
17/11/05 12:29:35 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/05 12:29:36 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/05 12:29:36 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/05 12:29:36 INFO FileOutputCommitter: Saved output of task 'attempt_20171105122936_0001_m_000001_0' to file:/tmp/KMeen.csv/_temporary/0/task_20171105122936_0001_m_000001
17/11/05 12:29:36 INFO FileOutputCommitter: Saved output of

Do you think thi will help you to invistigate the issue ?

Hi @Aalnafessah that might be a clue. Can you copy-paste your full stacktrace here? Also, are there any custom Hadoop settings in your Spark configuration, especially anything having to do with OutputCommiters? I'm not totally out of ideas yet, but I'm having a lot of trouble reproducing your issue in my own setups.

@ecurtin
I am using the default setting of spark. I have just enabled spark history sever in by adding the following lines in spark-defaults.conf:
spark.eventLog.enabled           true
#this is where to store event log
spark.eventLog.dir=/opt/spark/applicationHistory
#this is tell history server where to get event log
spark.history.fs.logDirectory=/opt/spark/applicationHistory

Hear is the the logs:
aa23916@picard03:~/SparkBench-New$ ./bin/spark-bench.sh examples/by-Ahmad/aalnafessah.conf
/homes/aa23916/SparkBench-New/lib
/homes/aa23916/SparkBench-New/lib
I'M HERE::: /homes/aa23916/SparkBench-New/lib/spark-bench-launch-2.1.1_0.2.2-RELEASE.jar
 *** SPARK-SUBMIT: [/opt/spark/bin/spark-submit, --class, com.ibm.sparktc.sparkbench.cli.CLIKickoff, --master, spark://picard03.doc.ic.ac.uk:7077, /homes/aa23916/SparkBench-New/lib/spark-bench-2.1.1_0.2.2-RELEASE.jar, /tmp/spark-bench-2836592916695796649.conf]
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/08 08:29:02 INFO SparkContext: Running Spark version 2.2.1-SNAPSHOT
17/11/08 08:29:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/08 08:29:03 INFO SparkContext: Submitted application: com.ibm.sparktc.sparkbench.cli.CLIKickoff
17/11/08 08:29:03 INFO SecurityManager: Changing view acls to: aa23916
17/11/08 08:29:03 INFO SecurityManager: Changing modify acls to: aa23916
17/11/08 08:29:03 INFO SecurityManager: Changing view acls groups to:
17/11/08 08:29:03 INFO SecurityManager: Changing modify acls groups to:
17/11/08 08:29:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(aa23916); groups with view permissions: Set(); users  with modify permissions: Set(aa23916); groups with modify permissions: Set()
17/11/08 08:29:03 INFO Utils: Successfully started service 'sparkDriver' on port 38806.
17/11/08 08:29:03 INFO SparkEnv: Registering MapOutputTracker
17/11/08 08:29:03 INFO SparkEnv: Registering BlockManagerMaster
17/11/08 08:29:03 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/11/08 08:29:03 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/11/08 08:29:03 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-81ee4ee7-e529-4c8d-99a9-07ec9ce62509
17/11/08 08:29:03 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
17/11/08 08:29:03 INFO SparkEnv: Registering OutputCommitCoordinator
17/11/08 08:29:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/11/08 08:29:04 INFO SparkUI: Bound SparkUI to <IP_ADDRESS>, and started at http://<IP_ADDRESS>:4040
17/11/08 08:29:04 INFO SparkContext: Added JAR file:/homes/aa23916/SparkBench-New/lib/spark-bench-2.1.1_0.2.2-RELEASE.jar at spark://<IP_ADDRESS>:38806/jars/spark-bench-2.1.1_0.2.2-RELEASE.jar with timestamp<PHONE_NUMBER>057
17/11/08 08:29:04 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://picard03.doc.ic.ac.uk:7077...
17/11/08 08:29:04 INFO TransportClientFactory: Successfully created connection to picard03.doc.ic.ac.uk/<IP_ADDRESS>:7077 after 27 ms (0 ms spent in bootstraps)
17/11/08 08:29:04 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20171108082904-0175
17/11/08 08:29:04 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20171108082904-0175/0 on worker-20171031134621-<IP_ADDRESS>-38385 (<IP_ADDRESS>:38385) with 20 cores
17/11/08 08:29:04 INFO StandaloneSchedulerBackend: Granted executor ID app-20171108082904-0175/0 on hostPort <IP_ADDRESS>:38385 with 20 cores, 1024.0 MB RAM
17/11/08 08:29:04 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20171108082904-0175/1 on worker-20171031134653-<IP_ADDRESS>-42778 (<IP_ADDRESS>:42778) with 16 cores
17/11/08 08:29:04 INFO StandaloneSchedulerBackend: Granted executor ID app-20171108082904-0175/1 on hostPort <IP_ADDRESS>:42778 with 16 cores, 1024.0 MB RAM
17/11/08 08:29:04 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20171108082904-0175/0 is now RUNNING
17/11/08 08:29:04 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20171108082904-0175/1 is now RUNNING
17/11/08 08:29:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40179.
17/11/08 08:29:04 INFO NettyBlockTransferService: Server created on <IP_ADDRESS>:40179
17/11/08 08:29:04 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/11/08 08:29:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, <IP_ADDRESS>, 40179, None)
17/11/08 08:29:04 INFO BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:40179 with 366.3 MB RAM, BlockManagerId(driver, <IP_ADDRESS>, 40179, None)
17/11/08 08:29:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, <IP_ADDRESS>, 40179, None)
17/11/08 08:29:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, <IP_ADDRESS>, 40179, None)
17/11/08 08:29:04 INFO EventLoggingListener: Logging events to file:/opt/spark/applicationHistory/app-20171108082904-0175
17/11/08 08:29:04 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/11/08 08:29:05 INFO SparkContext: Starting job: first at KMeansDataGen.scala:66
17/11/08 08:29:05 INFO DAGScheduler: Got job 0 (first at KMeansDataGen.scala:66) with 1 output partitions
17/11/08 08:29:05 INFO DAGScheduler: Final stage: ResultStage 0 (first at KMeansDataGen.scala:66)
17/11/08 08:29:05 INFO DAGScheduler: Parents of final stage: List()
17/11/08 08:29:05 INFO DAGScheduler: Missing parents: List()
17/11/08 08:29:05 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at KMeansDataGenerator.scala:60), which has no missing parents
17/11/08 08:29:05 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.1 KB, free 366.3 MB)
17/11/08 08:29:05 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1458.0 B, free 366.3 MB)
17/11/08 08:29:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:40179 (size: 1458.0 B, free: 366.3 MB)
17/11/08 08:29:05 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
17/11/08 08:29:05 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at KMeansDataGenerator.scala:60) (first 15 tasks are for partitions Vector(0))
17/11/08 08:29:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/11/08 08:29:06 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:48348) with ID 0
17/11/08 08:29:06 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, <IP_ADDRESS>, executor 0, partition 0, PROCESS_LOCAL, 4829 bytes)
17/11/08 08:29:06 INFO BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:36737 with 366.3 MB RAM, BlockManagerId(0, <IP_ADDRESS>, 36737, None)
17/11/08 08:29:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:36737 (size: 1458.0 B, free: 366.3 MB)
17/11/08 08:29:07 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1337 ms on <IP_ADDRESS> (executor 0) (1/1)
17/11/08 08:29:07 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/11/08 08:29:07 INFO DAGScheduler: ResultStage 0 (first at KMeansDataGen.scala:66) finished in 2.066 s
17/11/08 08:29:07 INFO DAGScheduler: Job 0 finished: first at KMeansDataGen.scala:66, took 2.258399 s
17/11/08 08:29:08 INFO BlockManagerInfo: Removed broadcast_0_piece0 on <IP_ADDRESS>:40179 in memory (size: 1458.0 B, free: 366.3 MB)
17/11/08 08:29:08 INFO BlockManagerInfo: Removed broadcast_0_piece0 on <IP_ADDRESS>:36737 in memory (size: 1458.0 B, free: 366.3 MB)
17/11/08 08:29:08 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/homes/aa23916/SparkBench-New/spark-warehouse/').
17/11/08 08:29:08 INFO SharedState: Warehouse path is 'file:/homes/aa23916/SparkBench-New/spark-warehouse/'.
17/11/08 08:29:09 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
17/11/08 08:29:10 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/11/08 08:29:10 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:39504) with ID 1
17/11/08 08:29:10 INFO BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:34575 with 366.3 MB RAM, BlockManagerId(1, <IP_ADDRESS>, 34575, None)
17/11/08 08:29:10 INFO SparkContext: Starting job: csv at SparkFuncs.scala:20
17/11/08 08:29:10 INFO DAGScheduler: Got job 1 (csv at SparkFuncs.scala:20) with 2 output partitions
17/11/08 08:29:10 INFO DAGScheduler: Final stage: ResultStage 1 (csv at SparkFuncs.scala:20)
17/11/08 08:29:10 INFO DAGScheduler: Parents of final stage: List()
17/11/08 08:29:10 INFO DAGScheduler: Missing parents: List()
17/11/08 08:29:10 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[4] at csv at SparkFuncs.scala:20), which has no missing parents
17/11/08 08:29:10 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 75.7 KB, free 366.2 MB)
17/11/08 08:29:10 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.0 KB, free 366.2 MB)
17/11/08 08:29:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:40179 (size: 30.0 KB, free: 366.3 MB)
17/11/08 08:29:10 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/11/08 08:29:10 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[4] at csv at SparkFuncs.scala:20) (first 15 tasks are for partitions Vector(0, 1))
17/11/08 08:29:10 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
17/11/08 08:29:10 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, <IP_ADDRESS>, executor 1, partition 0, PROCESS_LOCAL, 4829 bytes)
17/11/08 08:29:10 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, <IP_ADDRESS>, executor 0, partition 1, PROCESS_LOCAL, 4829 bytes)
17/11/08 08:29:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:36737 (size: 30.0 KB, free: 366.3 MB)
17/11/08 08:29:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:34575 (size: 30.0 KB, free: 366.3 MB)
17/11/08 08:29:11 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 1751 ms on <IP_ADDRESS> (executor 0) (1/2)
17/11/08 08:29:13 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 3339 ms on <IP_ADDRESS> (executor 1) (2/2)
17/11/08 08:29:13 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
17/11/08 08:29:13 INFO DAGScheduler: ResultStage 1 (csv at SparkFuncs.scala:20) finished in 3.342 s
17/11/08 08:29:13 INFO DAGScheduler: Job 1 finished: csv at SparkFuncs.scala:20, took 3.375767 s
17/11/08 08:29:13 INFO FileFormatWriter: Job null committed.
*** KMeans data gen *******
17/11/08 08:29:14 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
17/11/08 08:29:14 INFO CodeGenerator: Code generated in 234.107409 ms
17/11/08 08:29:14 INFO CodeGenerator: Code generated in 28.191852 ms
17/11/08 08:29:14 INFO SparkContext: Starting job: show at SparkFuncs.scala:21
17/11/08 08:29:14 INFO DAGScheduler: Got job 2 (show at SparkFuncs.scala:21) with 1 output partitions
17/11/08 08:29:14 INFO DAGScheduler: Final stage: ResultStage 2 (show at SparkFuncs.scala:21)
17/11/08 08:29:14 INFO DAGScheduler: Parents of final stage: List()
17/11/08 08:29:14 INFO DAGScheduler: Missing parents: List()
17/11/08 08:29:14 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[13] at show at SparkFuncs.scala:21), which has no missing parents
17/11/08 08:29:14 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 14.7 KB, free 366.2 MB)
17/11/08 08:29:14 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 6.9 KB, free 366.2 MB)
17/11/08 08:29:14 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:40179 (size: 6.9 KB, free: 366.3 MB)
17/11/08 08:29:14 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
17/11/08 08:29:14 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[13] at show at SparkFuncs.scala:21) (first 15 tasks are for partitions Vector(0))
17/11/08 08:29:14 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
17/11/08 08:29:14 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 3, <IP_ADDRESS>, executor 1, partition 0, PROCESS_LOCAL, 6767 bytes)
17/11/08 08:29:14 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:34575 (size: 6.9 KB, free: 366.3 MB)
17/11/08 08:29:15 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 3) in 286 ms on <IP_ADDRESS> (executor 1) (1/1)
17/11/08 08:29:15 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
17/11/08 08:29:15 INFO DAGScheduler: ResultStage 2 (show at SparkFuncs.scala:21) finished in 0.287 s
17/11/08 08:29:15 INFO DAGScheduler: Job 2 finished: show at SparkFuncs.scala:21, took 0.304497 s
+------+-------------+---------+----------+----------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------------------+----------------------+-----------------+--------------------+--------------------+-----------------+-----------------------+--------------------+--------------------+--------------------+--------------------+
|  name|    timestamp| generate|   convert|      save|total_runtime|scaling|numCols|              output|  k|input|numPartitions|numRows|run|spark.driver.host|spark.history.fs.logDirectory|spark.eventLog.enabled|spark.driver.port|          spark.jars|      spark.app.name|spark.executor.id|spark.submit.deployMode|        spark.master|  spark.eventLog.dir|        spark.app.id|         description|
+------+-------------+---------+----------+----------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------------------+----------------------+-----------------+--------------------+--------------------+-----------------+-----------------------+--------------------+--------------------+--------------------+--------------------+
|kmeans|1510129744908|304938235|4364358709|3988806456|<PHONE_NUMBER>|    0.6|      4|file:///tmp/KMeen...|  2|     |            2|     10|  0|    <IP_ADDRESS>|         /opt/spark/applic...|                  true|            38806|file:/homes/aa239...|com.ibm.sparktc.s...|           driver|                 client|spark://picard03....|/opt/spark/applic...|app-2017110808290...|*** KMeans data g...|
+------+-------------+---------+----------+----------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------------------+----------------------+-----------------+--------------------+--------------------+-----------------+-----------------------+--------------------+--------------------+--------------------+--------------------+

17/11/08 08:29:15 INFO SparkContext: Invoking stop() from shutdown hook
17/11/08 08:29:15 INFO SparkUI: Stopped Spark web UI at http://<IP_ADDRESS>:4040
17/11/08 08:29:15 INFO StandaloneSchedulerBackend: Shutting down all executors
17/11/08 08:29:15 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
17/11/08 08:29:15 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/11/08 08:29:15 INFO MemoryStore: MemoryStore cleared
17/11/08 08:29:15 INFO BlockManager: BlockManager stopped
17/11/08 08:29:15 INFO BlockManagerMaster: BlockManagerMaster stopped
17/11/08 08:29:15 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/11/08 08:29:15 INFO SparkContext: Successfully stopped SparkContext
17/11/08 08:29:15 INFO ShutdownHookManager: Shutdown hook called
17/11/08 08:29:15 INFO ShutdownHookManager: Deleting directory /tmp/spark-a3394d00-e542-4a28-be39-a51bf7b072e4
aa23916@picard03:~/SparkBench-New$

@Aalnafessah Thanks for the additional info! I have a new strong suspicion about the source of your issue.
My guess is that your files are being written out, but they're being written out to the local file system of the executors and not the driver. So if you have access to the worker nodes in your cluster, see if data has been written out to /tmp/KMeen.csv on those nodes.
Using HDFS would get around this, but I remember from earlier in our conversation that you had issues with HDFS. I think the issue with your HDFS setup is a separate problem from the problem with the local filesystem. From looking at some sources that have similar errors (Wrong FS expected: file:///...) it seems that the issue is generally related to setting the PATH for Hadoop settings.
Let me know if I'm right about the data getting written out locally to executors!

@ecurtin
Yes, you are correct. The master node has empty folder but I found the output CSV file in executers nodes under this path:
/tmp/KMeenFinal.csv/_temporary/0/task_20171108200154_0001_m_000000

and the file name in the executer1 is :
root@picard05:/tmp/KMeenFinal.csv/_temporary/0/task_20171108200154_0001_m_000000# ls
part-00000-85221cb0-5ee7-46ba-9fe0-c61dbc42f57a-c000.csv

After i run the KMean data generator successfully  ,  I tried to run the KMean work load with this conf file :
spark-bench = {
  spark-submit-config = [{
    spark-args = {
      master = "spark://picard03.doc.ic.ac.uk:7077"
    }
    workload-suites = [
    {
      descr = "*** KMeans WorkLoad  *******"
      benchmark-output = "console"
      workloads = [
       {
         name = "kmeans"
         input = "file:///tmp/KMeen.csv"
         k = 10
       }
      ]
    }
    ]
  }]

}

But i got this issue:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Unable to infer schema for CSV. It must be specified manually.;

How can it be specified manually  ?

@Aalnafessah I believe that issue is related to the first issue. I believe at least one node is picking up /tmp/KMeen.csv on its local filesystem and that file is actually empty (or, just has a SUCCESS message but no data) so when the KMeans process tries to figure out the schema, it can't because there is no data to figure on that particular node.
When I run the following locally on my laptop, the KMeans process is able to pick up the data and infer the schema.

spark-bench = {
  spark-submit-config = [{
    spark-args = {
      master = "local[*]"
    }
    suites-parallel = false
    workload-suites = [
      {
        descr = "Generating data"
        benchmark-output = "console"
        workloads = [
          {
            name = "data-generation-kmeans"
            rows = 10
            cols = 4
            output = "file:///tmp/KMeen.csv"
          }
        ]
      },
      {
        descr = "Using data"
        benchmark-output = "console"
        workloads = [
          {
            name = "kmeans"
            input = "file:///tmp/KMeen.csv"
            k = 10
          }
        ]
      }
    ]
  }]
}

With output
$ ./bin/spark-bench.sh examples/example.conf
 *** SPARK-SUBMIT: [/opt/spark-2.1.1-bin-hadoop2.7/bin/spark-submit, --class, com.ibm.sparktc.sparkbench.cli.CLIKickoff, --master, local[*], /Users/ecurtin/git/spark-bench/target/assembly/spark-bench-2.1.1_0.2.2-RELEASE.jar, {"spark-bench":{"spark-submit-config":[{"spark-args":{"master":"local[*]"},"suites-parallel":false,"workload-suites":[{"benchmark-output":"console","descr":"Generating data","workloads":[{"cols":4,"name":"data-generation-kmeans","output":"file:///tmp/KMeen.csv","rows":10}]},{"benchmark-output":"console","descr":"Using data","workloads":[{"input":"file:///tmp/KMeen.csv","k":10,"name":"kmeans"}]}]}]}}]
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/09 19:02:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/09 19:02:13 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/11/09 19:02:16 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/11/09 19:02:16 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/11/09 19:02:16 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/11/09 19:02:16 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/11/09 19:02:16 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/11/09 19:02:16 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/09 19:02:17 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/09 19:02:17 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/09 19:02:17 INFO FileOutputCommitter: Saved output of task 'attempt_20171109190217_0001_m_000001_0' to file:/tmp/KMeen.csv/_temporary/0/task_20171109190217_0001_m_000001
17/11/09 19:02:17 INFO FileOutputCommitter: Saved output of task 'attempt_20171109190217_0001_m_000000_0' to file:/tmp/KMeen.csv/_temporary/0/task_20171109190217_0001_m_000000
Generating data
+------+-------------+---------+----------+---------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+---------------+
|  name|    timestamp| generate|   convert|     save|total_runtime|scaling|numCols|              output|  k|input|numPartitions|numRows|run|spark.driver.host|spark.driver.port|hive.metastore.warehouse.dir|          spark.jars|      spark.app.name|spark.executor.id|spark.submit.deployMode|spark.master|       spark.app.id|    description|
+------+-------------+---------+----------+---------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+---------------+
|kmeans|1510272134042|214357246|2179297077|839766076|<PHONE_NUMBER>|    0.6|      4|file:///tmp/KMeen...|  2|     |            2|     10|  0|     <IP_ADDRESS>|            54562|        file:/Users/ecurt...|file:/Users/ecurt...|com.ibm.sparktc.s...|           driver|                 client|    local[*]|local-1510272133808|Generating data|
+------+-------------+---------+----------+---------+-------------+-------+-------+--------------------+---+-----+-------------+-------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+---------------+

17/11/09 19:02:17 INFO FileInputFormat: Total input paths to process : 2
17/11/09 19:02:17 INFO FileInputFormat: Total input paths to process : 2
17/11/09 19:02:18 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
17/11/09 19:02:18 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
Using data
+------+-------------+--------+---------+--------+----+-------------+----+-------------+------+---+--------------------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+-----------+
|  name|    timestamp|    load|    train|    test|save|total_runtime|seed|maxIterations|output|  k|               input|run|spark.driver.host|spark.driver.port|hive.metastore.warehouse.dir|          spark.jars|      spark.app.name|spark.executor.id|spark.submit.deployMode|spark.master|       spark.app.id|description|
+------+-------------+--------+---------+--------+----+-------------+----+-------------+------+---+--------------------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+-----------+
|kmeans|1510272138064|71890988|390076275|15317369|null|    477284632| 127|            2|      | 10|file:///tmp/KMeen...|  0|     <IP_ADDRESS>|            54562|        file:/Users/ecurt...|file:/Users/ecurt...|com.ibm.sparktc.s...|           driver|                 client|    local[*]|local-1510272133808| Using data|
+------+-------------+--------+---------+--------+----+-------------+----+-------------+------+---+--------------------+---+-----------------+-----------------+----------------------------+--------------------+--------------------+-----------------+-----------------------+------------+-------------------+-----------+

The best way to get around this issue that you're having is to use a HDFS or object storage. Using local filesystems is fine for single node (i.e. development laptops) but you're experiencing the limitations of relying on local filesystem paths: the data doesn't get actually get written to every node's filesystem, but it will write an empty file and then silently or weirdly fail, as you're experiencing. I highly recommend using HDFS to store data rather than the local filesystem. Behind the scenes, HDFS is using your local filesystem, but it provides a useful layer of abstraction.

I'm gonna go ahead and close this issue, but feel free to email me or open another issue if you have any questions :) And thank you for using Spark-Bench!


feat(*): add MapDeclaration to metamodel

Closes #

The implementation of the map declaration feature will be delivered incrementally through a phased approach. Fundamental to this is adding the MapDeclaration to the concerto metamodel. At its core, a MapDeclaration consists of one MapKeyType, and a general AggregateValueType which may in some cases be specialized to an AggregateRelationshipValueType
For more information on the design specification see here
Changes

 Add MapDeclaration, MapKeyType, AggregateValueType, AggregateRelationshipValueType to metamodel

Related Issues

Issue #447

Author Checklist

[x] Ensure you provide a DCO sign-off for your commits using the --signoff option of git commit.
[ ] Vital features and changes captured in unit and/or integration tests
[x] Commits messages follow AP format
[ ] Extend the documentation, if necessary
[ ] Merging to main from fork:branchname

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 94.03%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
83

Relevant Lines:
84

💛  - Coveralls


Erroneous data-to-data copy with other_subset

The bug can be reproduced by running test_out_success in tests/transformations/redundant_copy_test.py.
The test produces an SDFG with an edge from access node A to access node C. The edge has source subset 0, 0:3, 4 and destination subset 1, 2, 0:3, 4.
However, if you uncomment the assertion on line 91, it will fail.
The compiled program appears to copy the correct subset of A, but on C[0:3, 2, 0, 4] instead of C[1, 2, 0:3, 4].

Linked to the wrong PR

Seems like it was erroneously reopened.


Changes in next release

I've created a working document listing changes for the next release which will support the new Vega-Lite 4 schema. I will be using this as an opportunity to implement any necessary breaking changes to the API, but would like to keep these to a minimum.
Vega-Lite 4 is still under development, so this is a work in progress and may change to reflect further changes to the Vega-Lite schema.
This should be useful for @DougBurke in keeping the Haskell port hvega up to date.
Feel free to use this issue to comment on or propose any changes prior to the next release.

Nooooooo. More changes to follow....
:-)
Ta

When I created hvega I came up with the following type aliases (well, some have been added later, and you already have LabelledSpec) to help me understand the structure a bit better. I don't know if they would be of interest to you (where my VLSpec is a type alias for the Haskell JSON type) and Haskell's type is synonymous to Elm's type alias, I believe:
{-|

A convenience type-annotation label. It is the same as 'Data'.

@since <IP_ADDRESS>
-}
type PropertySpec = (VLProperty, VLSpec)

{-|

Represents a named Vega-Lite specification, usually generated by a
function in this module. You shouldn't need to create @LabelledSpec@
tuples directly, but they can be useful for type annotations.
-}
type LabelledSpec = (T.Text, VLSpec)

{-|

Represent those functions which can be chained together using function
composition to append new specifications onto an existing list.
-}
type BuildLabelledSpecs = [LabelledSpec] -> [LabelledSpec]

{-|

Convenience type-annotation label to indicate a color value.
There is __no attempt__ to validate that the user-supplied input
is a valid color.

Any supported HTML color specification can be used, such as:

@
\"#eee\"
\"#734FD8\"
\"crimson\"
\"rgb(255,204,210)\"
\"hsl(180, 50%, 50%)\"
@

@since <IP_ADDRESS>
-}

type Color = T.Text

{-|

Convenience type-annotation label to indicate an opacity value, which
lies in the range 0 to 1 inclusive. There is __no attempt__ to validate
that the user-supplied value falls in this range.

A value of 0 indicates fully transparent (see through), and 1 is
fully opaque (does not show anything it is on top of).

@since <IP_ADDRESS>
-}

type Opacity = Double

{-|

Convenience type-annotation label to indicate an angle, which is measured
in degrees from the horizontal (so anti-clockwise).

The value should be in the range 0 to 360, inclusive, but __no attempt__ is
made to enforce this.

@since <IP_ADDRESS>
-}

type Angle = Double

I have also created my own type for the zindex value, since it seems to really be 0 or 1 (although the docs are unclear whether other values have any possible meaning, and I could see it changing in the future):
{-|

At what "depth" (z index) is the item to be drawn (a relative depth
for items in the visualization).

@since <IP_ADDRESS>
-}

data ZIndex =
  ZBack
  -- ^ The item is drawn behind (this corresponds to the Vega Lite
  --   @0@ value).
  | ZFront
  -- ^ The item is drawn in front (this corresponds to the Vega Lite
  --   @1@ value).
  | ZValue Natural
  -- ^ This is provided in case there is any need to use a z index value
  --   other than @0@ or<EMAIL_ADDRESS>

Thanks for these suggestions. There is some debate in the Elm community as to the benefits of type aliasing (see, this thread for example).
I am not sure I have a consistent view on the matter. My instinct would be to include more type aliases like the examples you  provide, but it would appear that some people find the additional level of abstraction unhelpful.
I had a look again at why I chose to create the alias Data, DataColumn and DataRow given they are just ( VLProperty, Spec ), List LabelledSpec and Spec which are used by many other functions. In these cases I found that without the alias, it really wasn't clear which functions were creating data fields and which were modifying the properties of a specification. While underneath everything is a JSON spec, it seemed helpful to differentiate the semantic differences in these cases.
For other cases (and I'd argue opacity and angle fall into this category), the semantic meaning of a parameter seems clearer and more straightforward so I feel there is less benefit in the alias.
Color is an interesting case though, but for slightly different reasons. I can imagine that we might in the future have stronger typing for a color specification rather than relying on an arbitrary string. Perhaps there is an argument to abstract this in order to prepare the way such changes. On the same grounds, we might alias Expression for Vega expressions that are used by a number of functions (e.g. fiExpr and axLabelExpr and shortly hdLabelExpr).

On the z-index. There are a few cases where more than 1 and 0 may be needed. In the example below, three values of axis z-index are used to position y gridlines above x gridlines above circles. I don't think this could be achieved with z-indices of 1 and 0 only.
zOrderTest : Spec
zOrderTest =
    let
        data =
            dataFromColumns []
                << dataColumn "x" (nums [ 20, 10 ])
                << dataColumn "y" (nums [ 10, 20 ])
                << dataColumn "cat" (strs [ "a", "b" ])

        enc =
            encoding
                << position X [ pName "x", pQuant, pAxis [ axZIndex 2 ] ]
                << position Y [ pName "y", pQuant, pAxis [ axZIndex 1 ] ]
                << color [ mName "cat", mMType Nominal, mLegend [] ]

        cfg =
            configure
                << configuration (coAxis [ axcoGridWidth 8 ])
                << configuration (coAxisX [ axcoGridColor "red" ])
                << configuration (coAxisY [ axcoGridColor "blue" ])
    in
    toVegaLite [ cfg [], data [], enc [], circle [ maSize 5000, maOpacity 1 ] ]

Thanks for the zindex example - it's now part of the hvega documentation (well, it will be once I get around to releasing <IP_ADDRESS>!). I was reading the VegaLite schema documentation perhaps too literally, as it seemed to focus on 0 and 1 being the main values.
I've changed ZIndex back to being an alias for Natural.

One thing I have added is a 'MNoOutliers' constructor for 'MarkProperty' since it looked like the outliers are displayed by default for a boxplot mark, and 'MOutliers []' doesn't turn them off. I'll update this with an example from the new hvega tutorial I've put together (basically a copy of your walk through) once it's released.

Sorry I missed your comment about noOutliers. Have amended so empty lists for maOutliers,  maBox, maMedian and maRule remove their respective elements. Have not applied to maTicks as these do not appear by default.


fix crystal playground cannot yield with one parameter

Fix #4526

Fixed by https://github.com/crystal-lang/crystal/commit/b7e92377b250e7042654a2d6f7d8b5b03aea814a


context.in_treesitter_capture appears to be broken

FAQ

[X] I have checked the FAQ and it didn't resolve my problem.

Announcement

[X] I have checked Breaking change announcement.

Minimal reproducible full config
It is often preferred to disable cmp inside comments using treesitter; however, after a recent update, this appears to be broken.
local fn = vim.fn
local install_path = fn.stdpath('data') .. '/site/pack/packer/start/packer.nvim'

if fn.empty(fn.glob(install_path)) > 0 then
  PackerBoostrap = fn.system(
    { 'git', 'clone', '--depth', '1', 'https://github.com/wbthomason/packer.nvim', install_path }
  )
  vim.cmd [[packadd packer.nvim]]
end

return require('packer').startup {
  function(use)
    use{'hrsh7th/nvim-cmp', requires = {'hrsh7th/cmp-nvim-lsp'}}
  end
}

require('cmp').setup {
  enabled = function()
     --disable completion in comments
    local context = require 'cmp.config.context'
     --keep command mode completion enabled when cursor is in a comment
    if vim.api.nvim_get_mode().mode == 'c' then
      return true
    else
      return not context.in_treesitter_capture("comment")
    end
  end
}

Description
Typing anything at all results in the following error being echoed.
Error detected while processing CursorMovedI Autocommands for "*":
Error executing lua callback: ...te/pack/packer/start/nvim-cmp/lua/cmp/config/context.lua:
42: attempt to call local 'get_captures_at_pos' (a nil value)
stack traceback:
        ...te/pack/packer/start/nvim-cmp/lua/cmp/config/context.lua:42: in function 'in_tr
eesitter_capture'
        /home/emdash00/dotfiles/.config/nvim/plugin/cmp.lua:177: in function 'enabled'
        .../nvim/site/pack/packer/start/nvim-cmp/lua/cmp/config.lua:136: in function 'enab
led'
        ...re/nvim/site/pack/packer/start/nvim-cmp/lua/cmp/init.lua:317: in function 'call
back'
        ...ite/pack/packer/start/nvim-cmp/lua/cmp/utils/autocmd.lua:49: in function 'emit'

        ...ite/pack/packer/start/nvim-cmp/lua/cmp/utils/autocmd.lua:23: in function <...it
e/pack/packer/start/nvim-cmp/lua/cmp/utils/autocmd.lua:22>

Steps to reproduce
This is a problem for all files.
Expected behavior
Should just be able to type anything at all, but typing anything at all throws an error. This is a pretty major breaking change. It was working fine until I updated Neovim which caused this to break.
Actual behavior
Constant errors from Neovim happen if I try to type anything and usage becomes impossible until I comment out that line.
Additional context
No response

I realized I was on an prerelease of neovim. I hadn't updated it in a while. Updating it fixes the issue.


File importer plugin

REDMINE-17078

Coverage remained the same at ?% when pulling c9f92560eefba7b9105ee6c8f421d45545a6c1f5 on awaismbajwa:file-importer-plugin into 6036de75047f0828e5a5f1fc48049e9cb48c5390 on codevise:master.


refresh rate reverting back to 60Hz

Using ROG ALLY, I set the resolution refresh rate manually to a higher Hz but after exiting a game it reverts back to 60Hz and I need to set it manually again.

Add the exact refresh rate to the resolution in options.txt and reinstall. Remove 60hz and add your preferred refresh rate. Should default to that from then on.

Perfect, thanks!


Remove useless dependency h2-mvstore

Description
The h2-mvstore dependency is of no use to us, so we will remove the dependency.

coverage: 48.141% (-0.004%) from 48.145% when pulling 6261c741e79338e37029773e4198de31f64b77dd on remove_h2-mvstore into 8e0256658e0994476a19e9c48f2b96e25aacdd5f on master.


Missing white space for keyword argument and @overload tag.

For example:
def foo(x: 1)
end

get parsed as - (Object) foo(x: 1) in document, but
# @overload foo(x: 1)
def foo(x: 1)
end

get parsed as - (Object) foo(x:1) in document.

Thanks for reporting!


Jira Data Center support testing

Fixes #
Proposed Changes

@v-rucdu  Please review this PR.

Closing this PR as it was created for Testing.... Thanks!!!


Heroku

Hey wondering if anyone can assist me.
I have been trying to host the time off app on heroku.
I get an error 👎
Error: Please install mysql package manually
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at Object. (/app/bin/wwww:4:13)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
Now I know someone will see this and probably laugh.
However any assistance. I don't mind paying for someones time to help me get setup.
Email me
<EMAIL_ADDRESS>

Hi @deanouk23, I did not try the app on Heroku, but there is a pull request waiting to be merged in that claims to add the integration with it: https://github.com/timeoff-management/application/pull/129/files
Please consider looking at it


Update to permit multiple root dirs

Within ephys.py, multiple instances of grabbing the full session path with find_full_path from the element's __init__.py. See parallel PR for workflow-array-ephys for the same functionality. Notably, workflow now depends on the same find_full_path from element-data-loader.utils - should both reference the same instance of find-full-path or should they maintain independent versions of the same function?

I just actually realized that my suggested changes are in pull request #35.  Could you please review this pull request first?

I just actually realized that my suggested changes are in pull request #35
Unfortunately, I wasn't aware of this PR when you and I discussed me taking on this multiple root task. The changes there are more comprehensive and cover all features present here.


Scripture Reading Plans - Initial screens

Scripture Reading Plans feature allows an SAB app to include one or more reading plans that the user can select to help provide a consistent encouragement in their reading.  The overall feature is large and extensive.  This first issue will just cover:

Make sure all necessary files are copied to the PWA from SAB
Create an entry in the slide drawer that takes you to a Plans route.
Design for plan page that can contain 1, 2 or 3 tabs (for "My Plans", "Choose Plan", "Completed Plans"
Display a list of all plans in the choose plan tab of the plans page
Display plan detail page for an available plan that displays two tabs, the plan information tab and the plan calendar tab
A plan information tab that will allow the plan to be opened or continued
A plan calendar tab that displays a day by day list of the references for that day
A "My Plans" page that shows all open plans.
Display a plan detail page for an open play that shows the plan information tab, the plan calendar tab and the plan settings tab
The plan settings tab will be limited to just a "Plan Close" at this point

Copy plan files from SAB to PWA Data
All images and sfm files for the plans configured for this project are located in plans folder of the app within the SAB application.  It does not appear that any of these files are being copied to the PWA data folder at this time.  The plan configuration also needs to be copied to the config.js file via ConvertConfig.  In appdef.xml, there is a Plans section that will contain a Plan entry for each reading plan that has been configured.  A segment of an example configuration looks like the following:
    <plans>
        <features type="plans">
            <e name="default-reminder-time-1" value="07:30" />
            <e name="default-reminder-time-2" value="20:00" />
        </features>
        <plan id="PRO31" days="31">
            <title>
                <t lang="default">Proverbs in 31 days</t>
            </title>
            <filename>plan-proverbs-31.txt</filename>
            <image width="910" height="512">plan-proverbs-31.jpg</image>
        </plan>

Each Plan segment contains the following information:

id : The id associated with the plan
days : The number of days it takes to complete the plan which will also be the number of calendar entries that will be displayed
title : An array of the name of the reading plan.   Multiple entries may be entered for different interface languages
filename : The name of the file that contains the actual reading plan data
image: If present, the name of the file containing the image that will be displayed in several of the plan screens

Create a Plan entry in the slide navigation drawer
The entry for the plans displays below "Share App" (if present), in the section below  the annotations section and above settings.  A calendar icon is used for the display.

Plan Selection Page
The plan selection page appears when the user selects "Plans" in the slide navigation drawer.  This page contains up to three tabs.
The three tabs are for "My Plans", which displays active reading plans.  The second tab is "Choose Plan".  The third tab is for "Completed Plans".   A tab is displayed in the the tab bar at the top of the screen if one or more plans are available for that category.  If only one tab is active, then no bar is displayed.  So, the first time the Plans page is displayed after the app has been installed, the "Choose Plan" tab is displayed and the tab bar does not display.

If there is only one plan configured, then when that plan is active, the "My Plans" tab would be the only one displayed with no bar displaying.  If there are multiple tabs active, when Plans is entered from the navigation drawer, the "My Plans" tab is displayed if present, otherwise the "Choose Plan" tab is displayed

Each of the three tabs shows an entry for each of the plans for that category.  The one exception to this is that plans with a current status of "Completed" appear in both the "Completed Plans" tab and the "Choose Plans" tab.  The entry displays the image associated with the plan, if configured, and then the plan title in the current interface language or the default entry if not available with the number of days associated with plan underneath the title as shown above.
The "Completed Plans" tab will be discussed in another issue, since with this issue, it won't be possible for a plan to be completed.
A plan file is a file that contains a few pieces of startup information and then a series of "day" entries indicating the reference or references to be read for that date.  A copy of an example entry for a more complex reading plan is below:
\id Bible365
\title Bible in a Year: 365-Day Reading Plan
\descr Regular Bible reading is one of the most important habits you can develop. This daily Bible reading plan will give you all the benefits of a steady diet of God's Word as you read through Scripture in a year. The whole Bible is arranged into 365 daily readings, including an Old Testament passage, a New Testament passage, and a Psalm or Proerb for each day.

\day 1
\ref GEN 1
\ref GEN 2:1-17
\ref MAT 1:1-25
\ref PSA 1:1-6

\day 2
\ref GEN 2:18-25
\ref GEN 3
\ref GEN 4:1-16
\ref MAT 2:1-18
\ref PSA 2:1-12

The case insensitive items are as follows:

id: As before, the id for the plan.  It is not displayed to the user
title: The title for the plan
descr: The description of the plan.
img: The name of the image to display on the plan selection screens
day #: A day entry defines the reference(s) to be read on the specified plan session.
ref: one or more reference entry appears after a day entry.  The reference is associated with the day entry that preceeded it.  Note that the reference is in standard USFM reference format.

Database Information
This is for reference purposes.  The Android and iOS apps save the information about the plan states in 3 database tables.  The first is a plan setup table.  This contains the plans and associated features, most of which are not discussed in this issue (things like the day of weeks and reminder times associated with the plan).  The second table is a plan state which contains an entry (plan id, state, date) for each time the plan state changed and the state it changed to (available, in progress or completed).  When checking a plan, the most recent entry sets the state of the plan.  If no entry is present, the plan is "available".  The final table is a plan progress entry (plan id, day, entry index, date) which contains an entry for each reference that has been marked as "read".
Design details
This section is added as a little background as to how part of the apps application is designed.  They have a structure of a group of Plans.  Each Plan contains an array of PlanDays.  Each PlanDay contains an array of PlanEntries.  At startup, the arrays are first populated by the information in the plan files.  After that, the database is read to update those arrays with the current plan status (available, completed, in progress) and the plan progress.  As each PlanEntry is updated as read, if it is the last entry for that day, the day is marked as completed.  There is a function to check whether the PlanEntry will complete the PlanDay (the PlanEntry is the last incomplete entry in the day). There is also a function to check whether the PlanEntry is the last entry in the Plan (all PlanDays are complete except for the current PlanDay and it is the last incomplete entry in the PlanDay).  The PWA does not need to be implemented in exactly the same way, but the end of day and end of plan functions will be needed in some form for the remaining issue.
Plan Detail Page
There are two variations of the plan detail page: the one that displays when you select a plan from the "Choose Plan" tab and the one that displays when a plan is selected from the "My Plans" tab.  They are similar but have some differences.  The "Choose Plan" version has two tabs: a plan information tab and a calendar tab.  The "My Plans" version has 3 tags: it adds a settings tab.
Information Tab
The information tabs for the two modes are slightly different. They both contain the image associated with the plan (if configured), the plan title, the plan duration, and the description. The "My Plans" version contains a line with the date the plan was started and a "Continue" button, which displays the calendar tab for the plan.  The "Choose Plan" version is the same without the date started information and a "Start Plan" button in place of the continue button.

The "Start Plan" button marks the plan as active and displays the calendar tab.  The one difference to note is that the calendar is for an active plan (discussed below) and the back arrow on the calendar will take it to the "My Plans" tab, not the "Choose Plan" tab.
Calendar Tab
The Calendar tab displays the reading plan in a day by day format with each day showing the references for each day.  A checkmark is in front of the reference and is checked if that plan item has been read. (Information on how an item is marked as "read" will be in another issue.  The reference text is written in the format for the current book collection at the time the plan screen is displayed.  (For example: for a reference of MAT 1, if cuk is the current book collection, this is displayed Mateo 1.  If World English Bible is the current book collection, it would appear as Matthew 1)  For this one, there should probably be some database or store that contains an entry for each completed reference item  The bar scrolls right and left to view all of the days in the plan.  For the "My Plans" version of the tab, the tab is initially positioned to the current entry.  Tapping on one of the references takes you to that reference in the scripture.

The "Choose Plan" version of the same tab looks much the same, except no items are checked (since the plan is inactive) and tapping on a reference does nothing.
Settings Tab
The settings tab in the standard app contains 3 sections: reminders, set dates, and stop plans.  We don't believe the reminders is going to be available in PWA.  The set dates will be covered in a later issue and will not be present in the initial version created by this issue.  The Settings tab for this issue will simply contain the stop plan section.

Pressing the "Stop Plan" button causes a verification screen to be displayed.  Pressing "No" just clears the model and remains on the settings tab.  Pressing "Yes" causes the plan to be marked as "Available" (and thus in the "Choose Plans" category) and moves to the Information tab for the plan from Choose Plans (so showing a "Start Plan" button).  In app terms, it deleted the entries in the setup table and progress table but left the entries in the plan status table alone, just adding a new "Available" status entry to the table.  Thus it deleted all information about which references have been read and what days of weeks etc have been selected but it kept all the history as to whether the plan has been started, stopped or completed.
Discussions on setting entries as read, marking plans as complete, setting a calendar and available days of weeks for the plan and other items are covered in other issues.  The intent of this issue is to make the plan information available to the PWA, to provide access to the Plan menu screens, and to implement the start and stop plan functions.

@mikaylahubbard: the TypeScript data format looks like it could be like this:
    plans?: {
        features?: {
            [key: string]: string;
        };
        plan: {
            id: string;
            days: number;
            title: {
                [lang: string]: string;
            };
            filename: string;
            image?: {
                width: number;
                height: number;
                filename: string;
            }
        }[];
    };


Remove "Message Relay Module" section from the docs

Description:
Since admin services are no longer support with Micro-integrator, "Message Relay Module" section should be removed in https://ei.docs.wso2.com/en/latest/micro-integrator/setup/message_builders_formatters/message-relay/ page.
Suggested Labels:
Bug
Affected Product Version:
7.0.0 Micro-integrator

Doc already updated


Cesium uses up 10k RAM each time it runs a program

I have all my programs sitting in archive except for Cesium and Prgm A, to run the shell. I thought it was great, as I could save a ton of RAM by archiving everything. However, Cesium seems to harm more than help. Each time I run a program, 10 kilobytes more are mysteriously taken up. This only happens when running with Cesium, whether the program being run is archived or not. When I run it myself, using ASM (I only tested an Assembly program) there is no RAM being lost. The only way to fix this without losing Cesium is to archive Cesium and Prgm A and do a manual RAM clear. Why is this happening? It is taking away the point of using Cesium for me. I am using a TI-84 Plus CE.

which version of Cesium are you running? I just tested with 1.1.2 and did not experience this behavior

This is an old issue that has been fixed for quite some time. You should update


Migrate to AndroidX

Migrate to AndroidX

@xamarindevelopervietnam Thanks for the PR, have added some comments, please check

Hi @muhaym, i didn't see any comment.

Hi @muhaym, i didn't see any comment.

Forgot to Submit Review :D, Added Now


[Compilation]: General Improvements to the website

Describe the compilation of issues
Right now there are a fair amount of minor issues (features/refactors/bugs) in the issues board which should be compiled into a single issue for clarity sake. The through line of all these issues are that they are quality of life improvements to the website that will boost user experience in minor ways. All these minor improvements will lead to a major improvement in general. Additionally many of these issues pose to improve the simplicity and organization of the website.
Breakdown of what needs to be completed

[x] #50
[x] #48
[x] #42
[x] #30
[x] #28
[ ] #10
[x] #9
[ ] #8
[ ] #7
[ ] #5
[x] #4
[ ] #3
[x] #2
[ ] #1

Gonna deprecate this issue. A fair amount of the issues were fixed but a good chunk of these are gonna need to be considered in other sprints.


Create mail4u.txt

Tungnan University, New Taipei City, Taiwan

@ryan4javacode I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host mail4u.edu not found: 3(NXDOMAIN) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for understanding!


Improve error message for subscripted generic step output types.

Describe changes
I modified the step interface to raise an error during initialization if subscripted generics were specified as a step output, e.g., Outputs(a=List[str], b=Dict[str, str]) should be Outputs(a=List, b=Dict) instead.
Previous behavior:
Step would get created, but raise TypeError: Subscripted generics cannot be used with class and instance checks at runtime.
New behavior:
A more concise error is raised during step initialization, e.g.: StepInterfaceError: Subscripted generics cannot be used as step outputs. For step 'load_image_data', use images=typing.Dict instead of images=typing.Dict[str, PIL.Image.Image].
Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[ ] If my change requires a change to docs, I have updated the documentation accordingly.
[ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
[x] I have added tests to cover my changes.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[x] Other (add details above)

Changed this back to draft since the current solution still has various issues:

doesn't check single outputs
isn't compatible with Python 3.9
doesn't have tests
fails spelling and linting checks

Changed to draft again since I might have just figured out how to support subscripted returns, which would be much better than raising an error.

Closing in favor of #806.


⚠ Bofrost.at has degraded performance

In 3a0fe5d, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1325 ms

Resolved: Bofrost.at performance has improved in 2b538f3.


Raw Ender not working with pumps

Raw ender does not seem to be pumpable.  This has been observed with the RotaryCraft pump, as well as the ender-thermic pump from Extra Utilities (although it does replace raw ender with stone).
I was able to use the Expanded Redstone pump successfully, but this seems to be due to an exploit discovered with that pump: it will actually collect fluid flow blocks in addition to source blocks.
Because of these issues, and observations about the behavior of raw ender, I'm assuming that you don't have raw ender source blocks?  Is it possible to make it configurable to whether or not raw ender has a source block instead of the current flow mechanics?

Interesting. Raw Ender currently uses the standard Forge BlockFluidFinite, so it would be good if those mods added proper support for that. Here's how I do it.
I could make which fluid type it uses configurable, though.

I use the fluid registry lookup. FluidRegistry.lookupFluidForBlock or something like that. If that returns null. my machines ignore it. However, it also has to extend BlockFluidBase or BlockLiquid (the parent of the vanilla blocks) and be of metadata zero. Which fails for yours?

The only methods I overwrite of BlockFluidFinite are texture-related, so I use the default Forge implementation for everything else.
BlockFluidFinite uses metadata to determine how 'full' that particular blockspace is. Metadata 7 is 'full' (1 buckets worth), and metadata 0 is 1/8 'full' (125mb worth). There are no source blocks--each block is representative of a certain amount of fluid.
So, when consuming a BlockFluidFinite, you need to take that into account, and only consume the amount of fluid that exists in the block based on its metadata.
You might want to use the IFluidBlock.drain method: https://github.com/MinecraftForge/MinecraftForge/blob/1710ls/src/main/java/net/minecraftforge/fluids/BlockFluidFinite.java#L312-L321

My systems deal in blocks - and only some can even handle non-bucket amounts.

@ReikaKalseki, okay, then your pump and BlockFluidFinite will just not be compatible, but it still might be a good idea to use the standard interface. Something like:
if (block instanceof IFluidBlock && ((IFluidBlock) block).canDrain(world, x, y, z))
{
    IFluidBlock fluidBlock = (IFluidBlock) block;
    // dry run
    FluidStack drainedFluidStack = fluidBlock.drain(world, x, y, z, false);
    if (drainedFluidStack.amount == FluidContainerRegistry.BUCKET_VOLUME)
    {
        // do the actual draining
        return fluidBlock.drain(world, x, y, z, true);
    }
    return null;
}

Not perfect, and will still fail to handle most BlockFluidFinite blocks (given that the fluid will likely disperse), but that might be the most Forge-compatible way to do it.

That code is not compatible with vanilla fluids.

@ReikaKalseki, you may have seen it before I edited in the else if block. Vanilla fluids will need special handling.
I feel like you think I'm doing something wacky with my mod and I have to prove to you that 'my way' is worth supporting, but I'm not doing anything wacky at all.
It's totally up to you if you want to support BlockFluidFinite, but just be aware that it is not specific to my mod; it is native to Forge. So you need to decide whether or not you want to support a Forge thing, not a mod-specific thing. It's not my job to provide you a perfect implementation.

Many modded blocks also use the BlockLiquid class, so it would need to be more than just lava in water.
Also, given that both vanilla and 95% of mod fluids use the source block system, it is in fact the Finite system that would need special handling.

@ReikaKalseki, I feel like you're deliberately misinterpreting me at this point. I don't have any more to say, as nothing you're talking about has anything to do with code that I have written.
Look at the Forge source code and create an issue on their repository if you don't like it:

https://github.com/MinecraftForge/MinecraftForge/blob/1710ls/src/main/java/net/minecraftforge/fluids/IFluidBlock.java
https://github.com/MinecraftForge/MinecraftForge/blob/1710ls/src/main/java/net/minecraftforge/fluids/BlockFluidBase.java
https://github.com/MinecraftForge/MinecraftForge/blob/1710ls/src/main/java/net/minecraftforge/fluids/BlockFluidClassic.java
https://github.com/MinecraftForge/MinecraftForge/blob/1710ls/src/main/java/net/minecraftforge/fluids/BlockFluidFinite.java

@ReikaKalseki, I feel like you're deliberately misinterpreting me at this point. I don't have anything more to say, as nothing you're talking about has anything to do with code that I have written.

....?

....?

Perhaps I'm misinterpreting you? What's your goal here? Are you trying to collaborate with me to find a solution?

As I said, I will need to insert special handling...in several places.
I will say that I do not really approve of the existence of two separate and fundamentally incompatible systems.

Right, and as I said, I had nothing to do with either of those systems. I don't like it either, but that's what exists.

There, I added support for it on my four pumps, though one of them can only handle full buckets.

Nice!
Leaving this open because I think the config option is still a good idea.

Config option?

@ReikaKalseki, your work is done; you can safely ignore this issue now. Appreciate the quick fix.
I was talking about the config option suggested in the OP to make Raw Ender optionally use BlockFluidClassic instead of BlockFluidFinite:

Is it possible to make it configurable to whether or not raw ender has a source block instead of the current flow mechanics?

No longer applicable. Raw Ender has been completely rewritten as fluids have changed drastically.


Add Icons to Navbar

What would you like to share?
Hello, my name is A.A.D.Sailaja. I am a Tech Enthusiastic and i'm looking forward to contribute in open source projects. Here in this project i would like to add Icons to Navbar so that it will increase website Attractiveness
Can you please assign me this issue under gssoc'24
Additional information
No response
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@punyakrit can you assign me this issue under GSSOC'24

this is not required


compose: Allow multiple image selection; don't send immediately

Thanks @AkashDhiman for your PR #5125 and your prep work for this in #4590; those were quite helpful and informative. 🙂
Marked as a draft/blocked for a reason I'll explain, but that reason doesn't affect the first three commits—
ComposeMenu [nfc]: Use our own new type definition for attachments
ComposeMenu [nfc]: Fix name of insertAttachment prop to be plural
ComposeMenu [nfc]: Convert to a function component
—and indeed those three would be good to land as prep for removing the blocker.
The blocker is that we have basically the behavior of #5291, except with the insertion of image links instead of Jitsi links. On iOS, you take or select a photo to upload, and sometimes the message text input is unchanged when you expect it to get an inserted [Uploading …]() link.
Fixes: #2366
Fixes: #4540
Replaces: #5125

Chat thread about the resulting UI on Android:
https://chat.zulip.org/#narrow/stream/243-mobile-team/topic/Android.20select.20multiple.20photos/near/1423109

Thanks for the review! Revision pushed.

@gnprice, I've just rebased this atop #5635, so it's ready for another code review. I'm working on spinning up the simulator to make a screen recording now. 🙂

Just pushed a revision fixing a rebase error in the new type Attachment's definition (r-n-document-picker changed an optional string property to required with string | null).
Here's a video of the feature working on my iPhone:
https://user-images.githubusercontent.com/22248748/211920752-a6325fb5-5c08-4c13-b113-8b778ef2ebd0.mp4

Thanks for the review! Revision pushed.

Thanks! Looks good; merging.


Add files via upload

Please add my college Roorkee Institute of Technology, Roorkee.

@mukeshRar ritroorkee.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@mukeshRar ritroorkee.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@mukeshRar ritroorkee.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.


default .env with local overrides

Hi,
I'm unsure if this now the correct place for a feature request (the link in the issue template no longer seems to be working).
I've been using Vue.js for some web development and their build tool have some nice .env support which I think would be nice to have in pipenv.
So with the Vue.js webpack template you can have a .env file which you can commit to your source control which is nice to provide some default configurations to all developers. You can then override that .env with a .env.local which will merge over the top replacing any redeclared environment variable values. This one is obviously not tracked by source control.
Wondering if that's a feature which others would also like to see.
Thanks.

I don’t object this, but feel this is out of scope of Pipenv. Pipenv does not cover project templating (vue-cli does), This is probably better suited for an actual project template tool, such as Cookiecutter.

@uranusjr Yeah I see the scope creep, though I didn't mean to imply pipenv going down templating route.
My thinking was that pipenv was already aware of .env files anyway and I found it a nice quality of life in Vue.js development when bringing new developers into a project. e.g. the non-local version can contain all the possible environment variables so all options are obvious as well as comments can be in the default one to add some extra documentation.

-1


Remove break lines in commit messages

Breaklines in a csv report caused issues when importing the data to an
spreadsheet application like LibreOffice Calc or Microsoft Excel.

good deal, ill cut a release sometime today


Add Metrics to track blocks proposed and orphaned

Motivation

We would like to have more insight into how a Lodestar's local validator is performing with regards to block production.
Description

This tracks how many blocks have been:

Blocks proposed and included in canonical chain
Orphaned blocks

related to #5255
Update
Tracking missed blocks will involve some other changes which I will include in another PR.

Closing in favour of

https://github.com/ChainSafe/lodestar/pull/5448


seafile-client: 6.2.3 -> 6.2.4

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/seafile-client/versions.
meta.description for seafile-client is: '"Desktop client for Seafile, the Next-generation Open Source Cloud Storage"'.
Compare changes on GitHub

Checks done (click to expand)

built on NixOS
Warning: no invocation of /nix/store/lahs0l8pkbinsa1cpkacpvzzwdcg4xax-seafile-client-6.2.4/bin/seafile-applet had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/lahs0l8pkbinsa1cpkacpvzzwdcg4xax-seafile-client-6.2.4/bin/.seafile-applet-wrapped had a zero exit code or showed the expected version
0 of 2 passed binary check by having a zero exit code.
0 of 2 passed binary check by having the new version present in output.
found 6.2.4 with grep in /nix/store/lahs0l8pkbinsa1cpkacpvzzwdcg4xax-seafile-client-6.2.4
directory tree listing: https://gist.github.com/1950e965b57ef140895369d720a17310
du listing: https://gist.github.com/4b184a817976e0d04e0cda33e88b8cb7

Test this build with Cachix (click to expand)

One time setup in nixpkgs Git checkout:
cachix use r-ryantm
git remote add r-ryantm https://github.com/r-ryantm/nixpkgs.git

Test this build:
git fetch r-ryantm && git checkout faf30029ce77473440bf40428081f3c04967272f && nix-shell --pure -I nixpkgs=. -p seafile-client

cc @dotlambda for testing.

Success on x86_64-linux (full log)
Attempted: seafile-client
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/db0hlxawlgsynayfspq2mi5chb68z1ny-seafile-client-6.2.4
shrinking /nix/store/db0hlxawlgsynayfspq2mi5chb68z1ny-seafile-client-6.2.4/bin/.seafile-applet-wrapped
strip is /nix/store/gpc2wld1s0c6qzx9326cwn1wcx29xzsj-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/db0hlxawlgsynayfspq2mi5chb68z1ny-seafile-client-6.2.4/bin
patching script interpreter paths in /nix/store/db0hlxawlgsynayfspq2mi5chb68z1ny-seafile-client-6.2.4
checking for references to /build in /nix/store/db0hlxawlgsynayfspq2mi5chb68z1ny-seafile-client-6.2.4...
postPatchMkspecs
postPatchMkspecs
/nix/store/db0hlxawlgsynayfspq2mi5chb68z1ny-seafile-client-6.2.4

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: seafile-client
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Timed out, unknown build status on aarch64-linux (full log)
Attempted: seafile-client
Partial log (click to expand)
[6104/16620] CXX obj/third_party/pdfium/fxcodec/fxcodec_jumbo_4.o
[6105/16620] CXX obj/third_party/sfntly/sfntly/index_sub_table_format5.o
[6106/16620] CXX obj/third_party/pdfium/third_party/fx_agg/fx_agg_jumbo_1.o
[6107/16620] STAMP obj/third_party/leveldatabase/leveldatabase.inputdeps.stamp
[6108/16620] CXX obj/third_party/icu/bundled_icuuc/propname.o
[6109/16620] CXX obj/third_party/protobuf/protobuf_lite/int128.o
[6110/16620] CXX obj/third_party/sfntly/sfntly/horizontal_metrics_table.o
building of '/nix/store/ps20dnv5a4k7m5k801vxq2zqy66kmiq5-qtwebengine-5.11.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/0xrmfrxiw9rphaa5vh2rv2y1ggy6ll4s-seafile-client-6.2.4.drv': 1 dependencies couldn't be built
error: build of '/nix/store/0xrmfrxiw9rphaa5vh2rv2y1ggy6ll4s-seafile-client-6.2.4.drv' failed

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: seafile-client, seafile-shared
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: seafile-client, seafile-shared
Partial log (click to expand)
shrinking RPATHs of ELF executables and libraries in /nix/store/9gbsgmgmbnhr0m7xmpd37ikm8cxp33ir-seafile-client-6.2.4
shrinking /nix/store/9gbsgmgmbnhr0m7xmpd37ikm8cxp33ir-seafile-client-6.2.4/bin/.seafile-applet-wrapped
strip is /nix/store/553rihc190vsyy8b22iqcq25a6489h8y-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/9gbsgmgmbnhr0m7xmpd37ikm8cxp33ir-seafile-client-6.2.4/bin
patching script interpreter paths in /nix/store/9gbsgmgmbnhr0m7xmpd37ikm8cxp33ir-seafile-client-6.2.4
checking for references to /build in /nix/store/9gbsgmgmbnhr0m7xmpd37ikm8cxp33ir-seafile-client-6.2.4...
postPatchMkspecs
postPatchMkspecs
/nix/store/9gbsgmgmbnhr0m7xmpd37ikm8cxp33ir-seafile-client-6.2.4
/nix/store/i5vvljl0ciw2i4hda5l8fibcf9487bm0-seafile-shared-6.2.4

Timed out, unknown build status on aarch64-linux (full log)
Attempted: seafile-client, seafile-shared
Partial log (click to expand)
[7053/16620] CXX obj/third_party/webrtc/p2p/rtc_p2p/turnportfactory.o
[7054/16620] CXX obj/third_party/webrtc/rtc_base/rtc_base_generic/nethelpers.o
[7055/16620] CXX obj/third_party/webrtc/rtc_base/stringutils/stringutils.o
[7056/16620] CXX obj/third_party/webrtc/rtc_base/rtc_base_generic/sslfingerprint.o
[7057/16620] CXX obj/third_party/webrtc/rtc_base/rtc_base_generic/openssladapter.o
[7058/16620] CXX obj/third_party/webrtc/modules/rtp_rtcp/rtp_rtcp_format/rtpfb.o
[7059/16620] CXX obj/third_party/webrtc/rtc_base/rtc_base_generic/rtccertificategenerator.o
building of '/nix/store/mgdlgx0q1ggwqvvdidzj8g8a8jn2wxyy-qtwebengine-5.11.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/cs375msjswygwf1f5cw7wngdw6n8qg3d-seafile-client-6.2.4.drv': 1 dependencies couldn't be built
error: build of '/nix/store/cs375msjswygwf1f5cw7wngdw6n8qg3d-seafile-client-6.2.4.drv' failed


pull request for e2e test at 02/05/2017 03:56:07

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


scrollToBottom doesn't work width jQuery 1.6+ because of the new the .prop() method

Right now, I'm using this fix:
function scrollToBottom() {
	if (jQuery.fn.jquery > "1.6") {
		inner.prop({ scrollTop: inner.prop("scrollHeight") });
	}
	else {
		inner.attr({ scrollTop: inner.attr("scrollHeight") });
	}
};

Everything else seems to work fine width jQuery 1.7.1.

Can you make a patch please? I'll merge it.