Update locals.tf

Pull Request Objective
This piece of work is being tracked in this GitHub Issue.

Checklist

[!NOTE]
Each items should be checked. Skipping below checks could delay your PR review!

[x] I have reviewed the style guide and ensured that my code complies with it
[x] All checks have passed (or override label applied, if I've used the override-static-analysis label, I've explained why)
[x] I have self-reviewed my code
[x] I have reviewed the checks and can attest they're as expected

Additional Comments

This pull request resolves https://github.com/ministryofjustice/data-platform-support/issues/611


Error on game start: components[0].components[x].label: Must be 80 or fewer in length.

The first suspect is an exceedingly long answer from OpenTDB, however it would take a very long answer to trigger this.
No reproduction steps yet.

Fixed in v1.8.3


Implement support for generating manifests with recursive types in task_types.go

As an AME developer I don't want to maintain a separate pseudo task spec type, due to the lack of support for recursive types in the controller-gen tool. Right now in order to support a pipeline field in the TaskSpec, I have to maintain a separate type with almost identical fields. This also blocks me from supporting multiple layers of steps in the pipeline, which for example would be required for mixing parallel and sequential steps in a pipeline.
Acceptance criteria:

There is no longer a separate PipelineStep type.
The configuration required for a Task is moved to a separate template type which can be used for a single step task or in a pipeline.

Background:
We have not been able to get the manifest generation to work with recursive types, hence the need for a separate PipelineStep type. This separate type is essentially a duplicate of the TaskSpec type. If we can get recursive types to work when generating manifests, we will be able to avoid this duplication and simplify code various places in the project.
This will also make it easier to support more advanced pipeline features.
See task_types.go to find the types mentioned here.

After switching to kube-rs this issue is no longer relevant as the problem stated here does not exist.


Replace _all views with tables build using centroid()

Details
:warning: This will be a breaking change. :warning:
The three "all" views will be removed in favor of building tables during import.  Since osm2pgsql 1.7.1 there is a centroid() option that should remove the need for these views:

osm.vbuilding_all
osm.vpoi_all
osm.vshop_all

See: https://osm2pgsql.org/doc/manual.html#geometry-objects-in-lua

Available to test from the dev image.
docker pull rustprooflabs/pgosm-flex:dev


Task doesn't do anything...

Hi there,
I would love to use this plugin but I don't seem to be able to get it to run.
this is my grunt task:
		shopify: {
			options: {
				api_key: '<%= shpfy.api_key %>',
				password: '<%= shpfy.password %>',
				url: '<%= shpfy.url %>',
				theme: '<%= shpfy.theme %>',
				base: './shopify/',
			},
		},

this is my watch:
			shopify: {
				files: [
					'./shopify/assets/**',
					'./shopify/config/**',
					'./shopify/snippets/**',
					'./shopify/layout/**',
					'./shopify/templates/**',
				],
				tasks: [
					'shopify',
				],
			},

And this is the output I get:
>> File "shopify/assets/BSBLSC.0.0.1.min.css" changed.
Running "shopify" task

Done, without errors.

When I run grunt shopify:upload I get this:
Running "shopify:upload" task
>> [grunt-shopify] - Uploading "assets/BSBLSC.0.0.1.min.css"
>> [grunt-shopify] - File "assets/BSBLSC.0.0.1.min.css" uploaded.
>> [grunt-shopify] - Uploading "assets/BSBLSC.0.0.1.min.js"
>> [grunt-shopify] - File "assets/BSBLSC.0.0.1.min.js" uploaded.
>> [grunt-shopify] - Theme deploy complete.

Btw I am using a very fresh and updated node install. (Might be a new version that breaks compatibility?)
Any idea?

I found the reason why the watch failed:
            shopify: {
                files: [
                    'shopify/assets/**',
                    'shopify/config/**',
                    'shopify/snippets/**',
                    'shopify/layout/**',
                    'shopify/templates/**',
                ],
                tasks: [
                    'shopify',
                ],
            },

The path cannot have ./ prepended to it. Removing it worked.
Hope this helps someone else ;)

👏


[BUG] UPDATE on a Databricks (10.4) DELTA table leads to JVM crash

The Spark Executor JVM crashes when UPDATE command is run on a Delta table on Databricks 10.4. This does not appear to break on Apache Spark (3.2.1, at least).
Repro code
import org.apache.spark.sql.types._
import org.apache.spark.sql._

spark.conf.set("spark.rapids.sql.enabled", false)

val schema = StructType(Array(
      StructField("country", StringType),
      StructField("year", StringType),
      )
    )

val corpus_rows = Array(
    Row("china", "1920"),
    Row("japan", "1920"),
    Row("china", "2021"),
)

val rdd = sc.parallelize(corpus_rows)

spark.createDataFrame(rdd, schema).repartition(1).write.mode(SaveMode.Overwrite).format("delta").save("/tmp/myth/delta/corpus")

val read_df = spark.read.format("delta").load("/tmp/myth/delta/corpus")
read_df.createOrReplaceTempView("mycorpus")

// Crashy Update
spark.conf.set("spark.rapids.sql.enabled", true)
sql(" UPDATE mycorpus SET year=2022 WHERE year=2023 ").show // Crash!

Crash output:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f1d3d3ae8f5, pid=24326, tid=0x00007f1bcd7ff700
#
# JRE version: OpenJDK Runtime Environment (Zulu <IP_ADDRESS>-CA-linux64) (8.0_302-b08) (build 1.8.0_302-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.302-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x18b8f5]
#
# Core dump written. Default location: /home/ubuntu/spark-rapids/core or core.24326
#
# An error report file with more information is saved as:
# /home/ubuntu/spark-rapids/hs_err_pid24326.log
#Stage 8:>                                                          (0 + 1) / 1]
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
/databricks/spark/bin/spark-shell: line 47: 24326 Aborted                 (core dumped) "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"

Hmm. Looks like this has to do with the implied cast in the WHERE clause:
UPDATE mycorpus
SET year=2022
WHERE year=2023 ---> Should've been a STRING.

A JVM crash is not ideal.

Relevant stack frames for future investigation:
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  ai.rapids.cudf.Table.partition(JJI[I)[J+0
j  ai.rapids.cudf.Table.partition(Lai/rapids/cudf/ColumnView;I)Lai/rapids/cudf/PartitionedTable;+23
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.$anonfun$partitionInternalAndClose$6(Lcom/nvidia/spark/rapids/GpuHashPartitioningBase;Lai/rapids/cudf/ColumnVector;Lai/rapids/cudf/Table;)Lai/rapids/cudf/PartitionedTable;+6
j  com.nvidia.spark.rapids.GpuHashPartitioningBase$$Lambda$7341.apply(Ljava/lang/Object;)Ljava/lang/Object;+12
j  com.nvidia.spark.rapids.Arm.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+2
j  com.nvidia.spark.rapids.Arm.withResource$(Lcom/nvidia/spark/rapids/Arm;Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.$anonfun$partitionInternalAndClose$5(Lcom/nvidia/spark/rapids/GpuHashPartitioningBase;Lorg/apache/spark/sql/vectorized/ColumnarBatch;Lai/rapids/cudf/ColumnVector;)Lai/rapids/cudf/PartitionedTable;+12
j  com.nvidia.spark.rapids.GpuHashPartitioningBase$$Lambda$7340.apply(Ljava/lang/Object;)Ljava/lang/Object;+12
j  com.nvidia.spark.rapids.Arm.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+2
j  com.nvidia.spark.rapids.Arm.withResource$(Lcom/nvidia/spark/rapids/Arm;Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.$anonfun$partitionInternalAndClose$1(Lcom/nvidia/spark/rapids/GpuHashPartitioningBase;Lorg/apache/spark/sql/vectorized/ColumnarBatch;)Lai/rapids/cudf/PartitionedTable;+37
j  com.nvidia.spark.rapids.GpuHashPartitioningBase$$Lambda$7331.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
j  com.nvidia.spark.rapids.Arm.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+2
j  com.nvidia.spark.rapids.Arm.withResource$(Lcom/nvidia/spark/rapids/Arm;Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.partitionInternalAndClose(Lorg/apache/spark/sql/vectorized/ColumnarBatch;)Lscala/Tuple2;+13
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.$anonfun$columnarEval$2(Lcom/nvidia/spark/rapids/GpuHashPartitioningBase;Lorg/apache/spark/sql/vectorized/ColumnarBatch;Lai/rapids/cudf/NvtxRange;)Lscala/Tuple2;+2
j  com.nvidia.spark.rapids.GpuHashPartitioningBase$$Lambda$7330.apply(Ljava/lang/Object;)Ljava/lang/Object;+12
j  com.nvidia.spark.rapids.Arm.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+2
j  com.nvidia.spark.rapids.Arm.withResource$(Lcom/nvidia/spark/rapids/Arm;Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.$anonfun$columnarEval$1(Lcom/nvidia/spark/rapids/GpuHashPartitioningBase;Lorg/apache/spark/sql/vectorized/ColumnarBatch;Lai/rapids/cudf/NvtxRange;)[Lscala/Tuple2;+27
j  com.nvidia.spark.rapids.GpuHashPartitioningBase$$Lambda$7329.apply(Ljava/lang/Object;)Ljava/lang/Object;+12
j  com.nvidia.spark.rapids.Arm.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+2
j  com.nvidia.spark.rapids.Arm.withResource$(Lcom/nvidia/spark/rapids/Arm;Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object;+3
j  com.nvidia.spark.rapids.GpuHashPartitioningBase.columnarEval(Lorg/apache/spark/sql/vectorized/ColumnarBatch;)Ljava/lang/Object;+21
j  org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$.$anonfun$prepareBatchShuffleDependency$3(Lcom/nvidia/spark/rapids/GpuExpression;Lorg/apache/spark/sql/vectorized/ColumnarBatch;)Ljava/lang/Object;+2
j  org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$$$Lambda$7150.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
j  org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$$anon$1.partNextBatch()V+154
j  org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$$anon$1.hasNext()Z+21
j  org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(Lscala/collection/Iterator;)V+44
j  org.apache.spark.shuffle.ShuffleWriteProcessor.write(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/ShuffleDependency;JLorg/apache/spark/TaskContext;Lorg/apache/spark/Partition;)Lorg/apache/spark/scheduler/MapStatus;+46
j  org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$3(Lorg/apache/spark/scheduler/ShuffleMapTask;Lorg/apache/spark/ShuffleDependency;Lorg/apache/spark/rdd/RDD;JLorg/apache/spark/TaskContext;)Lorg/apache/spark/scheduler/MapStatus;+13
j  org.apache.spark.scheduler.ShuffleMapTask$$Lambda$5245.apply()Ljava/lang/Object;+20
j  com.databricks.spark.util.ExecutorFrameProfiler$.record(Ljava/lang/String;Ljava/lang/String;Lscala/Function0;)Ljava/lang/Object;+82
j  org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$1(Lorg/apache/spark/scheduler/ShuffleMapTask;Lorg/apache/spark/TaskContext;)Lorg/apache/spark/scheduler/MapStatus;+194
j  org.apache.spark.scheduler.ShuffleMapTask$$Lambda$5232.apply()Ljava/lang/Object;+8
j  com.databricks.spark.util.ExecutorFrameProfiler$.record(Ljava/lang/String;Ljava/lang/String;Lscala/Function0;)Ljava/lang/Object;+82
j  org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Lorg/apache/spark/scheduler/MapStatus;+14
j  org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Ljava/lang/Object;+2
j  org.apache.spark.scheduler.Task.doRunTask(JILscala/Option;)Ljava/lang/Object;+151
j  org.apache.spark.scheduler.Task.$anonfun$run$1(Lorg/apache/spark/scheduler/Task;JILorg/apache/spark/metrics/MetricsSystem;Lscala/collection/immutable/Map;Lscala/Enumeration$Value;Lscala/Option;)Ljava/lang/Object;+116
j  org.apache.spark.scheduler.Task$$Lambda$5187.apply()Ljava/lang/Object;+28
j  org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;Lscala/collection/immutable/Map;Lscala/Option;Lscala/Enumeration$Value;)Ljava/lang/Object;+23
j  org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Lorg/apache/spark/executor/Executor$TaskRunner;Lscala/runtime/BooleanRef;)Ljava/lang/Object;+43
j  org.apache.spark.executor.Executor$TaskRunner$$Lambda$5183.apply()Ljava/lang/Object;+8
j  org.apache.spark.util.Utils$.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object;+4
j  org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Lorg/apache/spark/executor/Executor$TaskRunner;)V+965
j  org.apache.spark.executor.Executor$TaskRunner$$Lambda$5162.apply$mcV$sp()V+4
J 11063 C1 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007f1d255a9d6c [0x00007f1d255a9d00+0x6c]
j  com.databricks.spark.util.ExecutorFrameProfiler$.record(Ljava/lang/String;Ljava/lang/String;Lscala/Function0;)Ljava/lang/Object;+82
j  org.apache.spark.executor.Executor$TaskRunner.run()V+38
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11

@mythrocks I tried to run the snippet above on Databrick 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12) don't see any crash. Is there any special config needed?
Can you verify that you still reproduce the crash, please?

Curious, why the stack showed it called GPU cudf code, but the repro code contains spark.conf.set("spark.rapids.sql.enabled", false) to disable the GPU?
Is it correct? @mythrocks

Curious, why the stack showed it called GPU cudf code, but the repro code contains spark.conf.set("spark.rapids.sql.enabled", false) to disable the GPU? Is it correct? @mythrocks

It is enabled just before running the update operation.
// Crashy Update
spark.conf.set("spark.rapids.sql.enabled", true)
sql(" UPDATE mycorpus SET year=2022 WHERE year=2023 ").show

Ah, right. I should have enabled the plugin. Thanks for catching it. I'll correct the description.
I'm trying to repro it at my end as well.

Here's what I'm getting from compute-sanitizer:
========= Program hit named symbol not found (error 500) on CUDA API call to cuGetProcAddress.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x218540]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x124d44]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame: [0x128e67]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame: [0x128f68]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame: [0x114df]
=========                in /lib/x86_64-linux-gnu/libpthread.so.0
=========     Host Frame: [0x176ca9]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame: [0x11fb97]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame: [0x16ab5f]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame:__device_stub__ZN6nvcomp13unsnap_kernelEPKPKvPKmPKPvS5_P14nvcompStatus_tPm(void const * const *, unsigned long const *, void* const *, unsigned long const *, nvcompStatus_t*, unsigned long*) [0xf57cc]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame:nvcomp::gpu_unsnap(void const * const *, unsigned long const *, void* const *, unsigned long const *, nvcompStatus_t*, unsigned long*, int, CUstream_st*) [0xf58c7]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame:nvcompBatchedSnappyDecompressAsync [0x1090f4]
=========                in /tmp/nvcomp1455830403426930150.so
=========     Host Frame:cudf::io::nvcomp::batched_decompress(cudf::io::nvcomp::compression_type, cudf::device_span<cudf::device_span<unsigned char const , unsigned long=18446744073709551615> const , unsigned long=18446744073709551615>, cudf::device_span<cudf::device_span<unsigned char, unsigned long=18446744073709551615> const , unsigned long=18446744073709551615>, cudf::device_span<cudf::io::decompress_status, unsigned long=18446744073709551615>, unsigned long, unsigned long, rmm::cuda_stream_view) [0x1ccec90]
=========                in /tmp/cudf6781276602044116984.so
=========     Host Frame:cudf::io::detail::parquet::reader::impl::decompress_page_data(hostdevice_vector<cudf::io::parquet::gpu::ColumnChunkDesc>&, hostdevice_vector<cudf::io::parquet::gpu::PageInfo>&) [0x1d884bf]
=========                in /tmp/cudf6781276602044116984.so
=========     Host Frame:cudf::io::detail::parquet::reader::impl::read(std::vector<std::vector<int, std::allocator<int>>, std::allocator<std::vector<int, std::allocator<int>>>> const &) [0x1d8ab76]
=========                in /tmp/cudf6781276602044116984.so
=========     Host Frame:cudf::io::detail::parquet::reader::read(cudf::io::parquet_reader_options const &) [0x1d8aed4]
=========                in /tmp/cudf6781276602044116984.so
=========     Host Frame:cudf::io::read_parquet(cudf::io::parquet_reader_options const &, rmm::mr::device_memory_resource*) [0x1cf6260]
=========                in /tmp/cudf6781276602044116984.so
=========     Host Frame:Java_ai_rapids_cudf_Table_readParquet [0x14b4c35]
=========                in /tmp/cudf6781276602044116984.so
=========     Host Frame: [0x7fe98d018527]
=========                in

I wonder if this points towards predicate pushdown in the Parquet reader.

Should that be crash in nvcomp code?

~Ah, right. I should have enabled the plugin. Thanks for catching it. I'll correct the description.~ Edit: Nope, the script is correct, as it is.
I'm trying to repro it at my end as well.
Edit: Yep, it's still crashing:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f193ee788f5, pid=6511, tid=0x00007f17b9bff700
#
# JRE version: OpenJDK Runtime Environment (Zulu <IP_ADDRESS>-CA-linux64) (8.0_302-b08) (build 1.8.0_302-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.302-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x18b8f5]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/ubuntu/work/spark-rapids/hs_err_pid6511.log
#Stage 5:>                                                          (0 + 1) / 1]
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
/databricks/spark/bin/spark-shell: line 47:  6511 Aborted                 (core dumped) "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"

Does this mean you can also repro it outside databricks ?

I tried that. But the UPDATE command is not supported in Apache Spark.

I tried locally (non-db env) and it passed with Spark3.3.0 + delta-2.1.0 + plugin-22.10.
But not sure if it ran into the path that causes the crash.
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.0
      /_/

Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 1.8.0_342)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._

scala> import org.apache.spark.sql._
import org.apache.spark.sql._

scala>

scala> spark.conf.set("spark.rapids.sql.enabled", false)

scala>

scala> val schema = StructType(Array(
     |       StructField("country", StringType),
     |       StructField("year", StringType),
     |       )
     |     )
schema: org.apache.spark.sql.types.StructType = StructType(StructField(country,StringType,true),StructField(year,StringType,true))

scala>

scala> val corpus_rows = Array(
     |     Row("china", "1920"),
     |     Row("japan", "1920"),
     |     Row("china", "2021"),
     | )
corpus_rows: Array[org.apache.spark.sql.Row] = Array([china,1920], [japan,1920], [china,2021])

scala>

scala> val rdd = sc.parallelize(corpus_rows)
rdd: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = ParallelCollectionRDD[0] at parallelize at <console>:30

scala>

scala> spark.createDataFrame(rdd, schema).repartition(1).write.mode(SaveMode.Overwrite).format("delta").save("/tmp/myth/delta/corpus")

scala>

scala> val read_df = spark.read.format("delta").load("/tmp/myth/delta/corpus")
read_df: org.apache.spark.sql.DataFrame = [country: string, year: string]

scala> read_df.createOrReplaceTempView("mycorpus")

scala>

scala> // Crashy Update

scala> spark.conf.set("spark.rapids.sql.enabled", true)

scala> sql(" UPDATE mycorpus SET year=2022 WHERE year=2023 ").show
22/09/09 05:15:26 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU
22/09/09 05:15:26 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU
22/09/09 05:15:27 WARN GpuOverrides:
*Exec <HashAggregateExec> will run on GPU
  *Exec <ShuffleExchangeExec> will run on GPU
    *Partitioning <HashPartitioning> will run on GPU
    *Exec <HashAggregateExec> will run on GPU
      !Exec <FilterExec> cannot run on GPU because not all expressions can be replaced
        !Expression <ScalaUDF> UDF() cannot run on GPU because neither UDF implemented by class org.apache.spark.sql.delta.commands.UpdateCommand$$Lambda$6596/554794768 provides a GPU implementation, nor the conf `spark.rapids.sql.rowBasedUDF.enabled` is enabled
        *Exec <ProjectExec> will run on GPU
          *Expression <Alias> input_file_name() AS input_file_name()#766 will run on GPU
            *Expression <InputFileName> input_file_name() will run on GPU
          *Exec <FilterExec> will run on GPU
            *Expression <And> (isnotnull(year#645) AND (cast(year#645 as int) = 2023)) will run on GPU
              *Expression <IsNotNull> isnotnull(year#645) will run on GPU
              *Expression <EqualTo> (cast(year#645 as int) = 2023) will run on GPU
                *Expression <Cast> cast(year#645 as int) will run on GPU
            *Exec <FileSourceScanExec> will run on GPU

22/09/09 05:15:27 WARN GpuOverrides:
! <LocalTableScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.LocalTableScanExec
  @Expression <AttributeReference> path#778 could run on GPU
  @Expression <AttributeReference> partitionValues#779 could run on GPU
  @Expression <AttributeReference> size#780L could run on GPU
  @Expression <AttributeReference> modificationTime#781L could run on GPU
  @Expression <AttributeReference> dataChange#782 could run on GPU
  @Expression <AttributeReference> stats#783 could run on GPU
  @Expression <AttributeReference> tags#784 could run on GPU

22/09/09 05:15:27 WARN GpuOverrides:
!Exec <CollectLimitExec> cannot run on GPU because the Exec CollectLimitExec has been disabled, and is disabled by default because Collect Limit replacement can be slower on the GPU, if huge number of rows in a batch it could help by limiting the number of rows transferred from GPU to CPU. Set spark.rapids.sql.exec.CollectLimitExec to true if you wish to enable it
  @Partitioning <SinglePartition$> could run on GPU
  *Exec <ProjectExec> will run on GPU
    *Expression <Alias> cast(num_affected_rows#649L as string) AS num_affected_rows#816 will run on GPU
      *Expression <Cast> cast(num_affected_rows#649L as string) will run on GPU
    ! <CommandResultExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.CommandResultExec
      @Expression <AttributeReference> num_affected_rows#649L could run on GPU

+-----------------+
|num_affected_rows|
+-----------------+
|                0|
+-----------------+

Thanks ttnghia, I will try it on DB.

Found the root cause. See https://github.com/rapidsai/cudf/issues/11700

Here is the fix https://github.com/rapidsai/cudf/pull/11702

@firestarman do you have the stacktrace where we're trying to partition an empty table?  There's checks in the plugin where we have avoided trying to shuffle empty tables, curious what the call stack is and how we allowed an empty table to slip through.  The fix in cudf is still desired, just wondering how we got here.

The stacktrace that Mithun posted above should be somewhat relevant to the partitioning issue.
I'm also curious why this is caused by a partitioning issue instead of casting, because changing the query from year=2023 into year='2023' doesn't cause any crash.

I tried locally (non-db env) and it passed with Spark3.3.0 + delta-2.1.0 + plugin-22.10.

Sorry I'm late to this party, but I can confirm that this bug is specific to Databricks (though I'm not sure why). IIRC, I don't remember seeing the problem on Apache Spark + Delta.

The stacktrace that Mithun posted above should be somewhat relevant to the partitioning issue.

Ah, thanks!  I didn't look far enough back in the issue, missed that.

I'm also curious why this is caused by a partitioning issue instead of casting, because changing the query from year=2023 into year='2023' doesn't cause any crash.

I'm guessing because the subquery somehow ends up selecting nothing without it being a string to search for, and that causes an empty table to try to be partitioned.  Why it ends up selecting nothing is unclear though, as I would expect the query to either fail to compile due to a type mismatch or automatically apply a cast which should have worked.

Why it ends up selecting nothing is unclear though... ends up selecting nothing without it being a string to search for...

I'll have to work out the vagaries of how UPDATE works, but note that year=2023 should select no rows in the corpus, string or otherwise:
val corpus_rows = Array(
    Row("china", "1920"),
    Row("japan", "1920"),
    Row("china", "2021"),
)

I tried locally (non-db env) and it passed with Spark3.3.0 + delta-2.1.0 + plugin-22.10.

Sorry I'm late to this party, but I can confirm that this bug is specific to Databricks (though I'm not sure why). IIRC, I don't remember seeing the problem on Apache Spark + Delta.

I got a case to reproduce it locally.

Ah, thanks! I didn't look far enough back in the issue, missed that.

I'm also curious why this is caused by a partitioning issue instead of casting, because changing the query from year=2023 into year='2023' doesn't cause any crash.

I'm guessing because the subquery somehow ends up selecting nothing without it being a string to search for, and that causes an empty table to try to be partitioned. Why it ends up selecting nothing is unclear though, as I would expect the query to either fail to compile due to a type mismatch or automatically apply a cast which should have worked.

Here is the plan tree that ran into this crash.
scala> sql(" UPDATE mycorpus SET year=2022 WHERE year=2021 ")
22/09/15 00:37:56 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU
22/09/15 00:37:56 WARN GpuOverrides:
*Exec <HashAggregateExec> will run on GPU
  *Exec <ShuffleExchangeExec> will run on GPU
    *Partitioning <HashPartitioning> will run on GPU
    *Exec <HashAggregateExec> will run on GPU
      *Exec <ProjectExec> will run on GPU
        *Expression <Alias> input_file_name() AS input_file_name()#848 will run on GPU
          *Expression <InputFileName> input_file_name() will run on GPU
        *Exec <FilterExec> will run on GPU
          *Expression <ScalaUDF> UDF() will run on GPU
          *Exec <FilterExec> will run on GPU
            *Expression <And> (isnotnull(year#514) AND (cast(year#514 as int) = 2021)) will run on GPU
              *Expression <IsNotNull> isnotnull(year#514) will run on GPU
              *Expression <EqualTo> (cast(year#514 as int) = 2021) will run on GPU
                *Expression <Cast> cast(year#514 as int) will run on GPU
            *Exec <FileSourceScanExec> will run on GPU

This would be for the subquery in the update command
If changing year=2023 to year='2023', the query will not run into the crash path. (I added some logs and they were not printed out.)

@firestarman do you have the stacktrace where we're trying to partition an empty table? There's checks in the plugin where we have avoided trying to shuffle empty tables, curious what the call stack is and how we allowed an empty table to slip through. The fix in cudf is still desired, just wondering how we got here.

There is a defect in the empty batches check, and I made a PR https://github.com/NVIDIA/spark-rapids/pull/6564 to fix it.


ui.combobox returns different types of values

sometimes the return value is string (when selected by dropdown), sometimes, if you type some text in it, list is returned instead, for the same instance.
You can also use this tutorial to reproduce https://wave.h2o.ai/docs/examples/combobox

@MartinBarus I managed to reproduce it on Wave 0.22 but not on master (with the latest changes, next release).
It looks like it was caused by #1327 and has been unintentionally fixed by #1538
0.22
https://user-images.githubusercontent.com/15820796/189083615-f3600c82-09fa-4c00-a262-3b36e094c5b8.mov
After PR 1538  - latest changes
https://user-images.githubusercontent.com/15820796/189083428-6d3f8a1f-b035-4a2c-b531-a03071bd14fe.mov
Currently checking what was the bug and will write some tests to ensure it won't happen again.
cc: @mturoci

@aalencar I managed to repro it yesterday on master. The problem is that your combobox implementation relies on hitting enter at the end of typing. Instead, we should update wave.args continually as user types (same behavior as textbox).
Repro steps should be something like:

Run combobox.py example.
Write something to the first combobox.
Click submit button.
See the list in q.args.

I checked wrong branch, please ignore my last comment.


Won't ingest data

I have Concierge installed in an Ubuntu 22.04 VM and while it runs, it won't load any data from the web or locally. It just stays on 0%.

I recommend updating to the development branch and trying again.

Made a completely fresh Ubuntu 22.04 VM, installed the dev branch with no issues, it runs with no errors, but nothing is listening on port 8000. Looking at lsof -Pi I have several docker containers listening on some higher ports, and occasionally one pops up listening on port 8000, but disappears instantly before I can connect.

When running the install and launch, did you install_dev.py and launch_dev.py?

you shouldn't have to make a new VM - just run the proper python scripts

I'll try that now.

matt@matt-virtual-machine:~/Desktop/concierge$ sudo python3 launch_dev.py
[sudo] password for matt:
Checking Docker container status...
Ollama running
OpenSearch not found
Docker container dependencies don't appear to be running properly.
Start docker containers with CPU or GPU? [CPU] or GPU:
[+] Running 3/0
✔ Container ollama                 Running                                0.0s
✔ Container opensearch-dashboards  R...                                   0.0s
✔ Container opensearch-node1       Runnin...                              0.0s
INFO:     Will watch for changes in these directories: ['/home/matt/Desktop/concierge/concierge_shiny']
INFO:     Uvicorn running on http://<IP_ADDRESS>:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [7218] using WatchFiles
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/matt/Desktop/concierge/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
target(sockets=sockets)
File "/home/matt/Desktop/concierge/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run
return asyncio.run(self.serve(sockets=sockets))
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/matt/Desktop/concierge/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve
await self._serve(sockets)
File "/home/matt/Desktop/concierge/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve
config.load()
File "/home/matt/Desktop/concierge/lib/python3.10/site-packages/uvicorn/config.py", line 434, in load
self.loaded_app = import_from_string(self.app)
File "/home/matt/Desktop/concierge/lib/python3.10/site-packages/uvicorn/importer.py", line 22, in import_from_string
raise exc from None
File "/home/matt/Desktop/concierge/lib/python3.10/site-packages/uvicorn/importer.py", line 19, in import_from_string
module = importlib.import_module(module_str)
File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/matt/Desktop/concierge/concierge_shiny/app.py", line 4, in
from collection_management import collection_management_ui, collection_management_server
File "/home/matt/Desktop/concierge/concierge_shiny/collection_management.py", line 15, in
from ingester import ingester_ui, ingester_server
File "/home/matt/Desktop/concierge/concierge_shiny/ingester.py", line 6, in
from concierge_backend_lib.loading import load_file
File "/home/matt/Desktop/concierge/concierge_backend_lib/loading.py", line 9, in
from loaders.text import TextFileLoader
File "/home/matt/Desktop/concierge/loaders/text.py", line 3, in
from binaryornot.check import is_binary
ModuleNotFoundError: No module named 'binaryornot'

Oops, looks like I forgot to include a dependency there! I will fix the dev branch.
We have just released v0.3.0 which shouldn't have those issues, maybe you can try removing your current install and following the instructions on the main branch. Please let me know if you're still having trouble.

Np! I tried again, first it made me install cargo, which I did, now getting this:
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Downloading ruff-0.0.15.tar.gz (51 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.7/51.7 KB 957.3 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
ERROR: Package 'script-builder' requires a different Python: 3.10.12 not in '>=3.12'
Traceback (most recent call last):
File "/home/matt/concierge/dev_installer.py", line 1, in
from launch_concierge.concierge_installer.arguments import install_arguments
ModuleNotFoundError: No module named 'launch_concierge'

No problem. FYI just tried it, first it made me install cargo, which I did, then it gave me this:
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Downloading ruff-0.0.15.tar.gz (51 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.7/51.7 KB 957.3 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
ERROR: Package 'script-builder' requires a different Python: 3.10.12 not in '>=3.12'
Traceback (most recent call last):
File "/home/matt/concierge/dev_installer.py", line 1, in
from launch_concierge.concierge_installer.arguments import install_arguments
ModuleNotFoundError: No module named 'launch_concierge'

A couple of things:

That error is telling you that you have Python 3.10.12 but we only support 3.12 or higher, this is listed as a requirement in the project README.
Do you need the dev environment? If you're not looking to contribute code you can just pull the install script with pip using the instructions here: https://github.com/InfoSecInnovations/concierge?tab=readme-ov-file#quick-install. you no longer need to pull the whole repository.

I think cargo is a dependency of ruff, which is only used in the dev version.

I initially missed the Python requirement, looks like I may be waiting until Ubuntu 24.04 actually works in a VM :) Thank you very much for the help!

I've downgraded the requirements of the installer packages because I think they should probably work with Python 3.10 even though we're on a later version for development.
Could you try:
pip install launch-concierge --upgrade
python -m launch_concierge.install
and let me know if this works?
Thanks.


enhance  pixmap api

Issue details
enhance  pixmap api . such as awt Image api to draw string、Rotate and flip
Reproduction steps/code

@Override
    public void drawChars(char[] data, int offset, int length, int x, int y, int anchor) {
        // TODO hao to implement
    }

    @Override
    public void drawImage(Image img, int x, int y, int anchor) {
        if(!(img instanceof GdxImage gdxImage)) return;
        pixmap.drawPixmap(gdxImage.getPixmap(), x, y);
    }

Version of libGDX and/or relevant dependencies
1.13.0
Stacktrace
null
Please select the affected platforms

[ ] Android
[ ] iOS
[ ] HTML/GWT
[x] Windows
[ ] Linux
[ ] macOS

This could be tricky, as libGDX's API aims to be platform-agnostic. Maybe making a library for this is more likely.

Yeah, see if you learn anything from https://github.com/badboy-tian/nativefont for example.


Improve error reporting for unsupported XSD elements

Unsupported elements in XSDs lead to scala MatchError, without detail. This just throws an exception with some clue about what was not matched in several places.

Codecov Report

Merging #557 (7757056) into master (ccbb866) will decrease coverage by 0.35%.
The diff coverage is 0.00%.

:exclamation: Current head 7757056 differs from pull request most recent head 3cc1064. Consider uploading reports for the commit 3cc1064 to get more accurate results

@@            Coverage Diff             @@
##           master     #557      +/-   ##
==========================================
- Coverage   86.05%   85.69%   -0.36%
==========================================
  Files          18       18
  Lines         954      958       +4
  Branches       95       91       -4
==========================================
  Hits          821      821
- Misses        133      137       +4

Impacted Files
Coverage Δ

.../main/scala/com/databricks/spark/xml/package.scala
21.42% <ø> (ø)

...la/com/databricks/spark/xml/util/XSDToSchema.scala
62.02% <0.00%> (-3.31%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ccbb866...3cc1064. Read the comment docs.


Mark activate_pool operations as successful

Codecov Report

Merging #498 (a3a85af) into main (d90d176) will not change coverage.
The diff coverage is 100.00%.

:exclamation: Current head a3a85af differs from pull request most recent head 22783fd. Consider uploading reports for the commit 22783fd to get more accurate results

@@            Coverage Diff            @@
##              main      #498   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files          267       267
  Lines        15240     15249    +9
=========================================
+ Hits         15240     15249    +9

Impacted Files
Coverage Δ

internal/batchpin/batchpin.go
100.00% <ø> (ø)

internal/broadcast/manager.go
100.00% <ø> (ø)

internal/contracts/manager.go
100.00% <ø> (ø)

pkg/fftypes/transport_wrapper.go
100.00% <ø> (ø)

internal/assets/token_pool.go
100.00% <100.00%> (ø)

internal/assets/token_transfer.go
100.00% <100.00%> (ø)

internal/database/sqlcommon/operation_sql.go
100.00% <100.00%> (ø)

internal/dataexchange/dxhttps/dxhttps.go
100.00% <100.00%> (ø)

internal/events/dx_callbacks.go
100.00% <100.00%> (ø)

internal/events/operation_update.go
100.00% <100.00%> (ø)

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d90d176...22783fd. Read the comment docs.


[Feature request] Support Tailwind 4

Is your feature request related to a problem? Please describe.
TailwindCSS 4 is in alpha and will be released soon. Currently I cannot use this module with Tailwind 4 because package.json specifies ^3.4.0.
Describe the solution you'd like
A new release that supports both ^3 and ^4.
Describe alternatives you've considered
I tried updating tailwindcss to 4.0.0-alpha.9 in peerDependencies and devDependencies, but npm install failed because of other dependencies:

@tailwindcss/aspect-ratio
@tailwindcss/forms
@tailwindcss/line-clamp
@tailwindcss/typography

These all depend on tailwindcss version ^3.
I'm not sure when/if those plugins will be updated for Tailwind 4.
Additional context
Will this need some help from the maintainers of the 4 modules above?

Making the eslint-plugin-tailwindcss compatible with Tailwind CSS v4 won't be easy and it'll definitively take a lot of time. One of the main challenge will be to workaround the possible the absence of tailwind.config.js file.
Parsing of existing CSS file (containing the CSS vars) is also very slow compared to native config...
I'll work on it but keep in mind this project is free and I can only work on it on my free time 😉

Note that v4 is not planned to be stable until summer (source so this shouldn't be a huge priority.

@francoismassart in their announcement post about v4, they mention that configuration would still be available with a tailwind.config.js file, and it's something on their roadmap.

Support for JavaScript configuration files — reintroducing compatibility with the classic tailwind.config.js file to make migrating to v4 easy.

https://tailwindcss.com/blog/tailwindcss-v4-alpha#roadmap-to-v4-0
So in case parsing CSS proves to be too difficult, inefficient or time-consuming, would you be open to the idea of having such file mandatory in order to use eslint-plugin-tailwindcss?
To be totally honest here, I am not completely sold on the idea of defining all colors and other variables in a CSS file to begin with, so who knows, I can see people that will prefer to stick with the good old config file, especially if config-less Tailwind breaks other tools such as this one.
The only worrying part here is that they mention they are doing this to make v4 migrations easy, which may or may not imply they plan to get rid of it altogether in some later version, but I could be wrong.

I started reading the next branch of Tailwind CSS's codebase and I'll work on it soon on my free time, I've a full time job to comply to 😉

Quick update on v4 of eslint-plugin-tailwindcss:

It'll use the following stack:

Turborepo with:

eslint-plugin-tailwindcss as a package
several playgrounds as apps (React.js, Vue, etc.) for easier live testing

TypeScript
Vitest
@typescript-eslint
eslint-doc-generator

It'll only support Tailwind CSS v4, we'll drop support for v3
All rules will be rewritten and the strategy is to be based as much as possible on Tailwind CSS exposed methods:

better performances
stable results
custom logic will be added only when required

I don't have a lot of time to work on it as it is a side project, you can contribute:

Become a sponsor on GitHub or via thanks.dev
Make pull requests or provide exhaustive issues or feedbacks
Share the project on social medias
Lastly, the project is used by almost 40K projects, yet it has only 1.3K stars.
I hope reaching at least 4K stars might offer me the free usage of Copilot which I do not have.

Hello 👋,
Find some trouble with tailwind CLI, when use last alpha version with NPM.
audit fix --force
Installed latest tailwind 3 CLI.
Use it with astro js framework and node(18-20), maybe this is a reason, I don't know.
And I don't see this bug, If I am using Bun with bunx --bun command...
But in both ways, See problem with output.css file in vertical alight ... I don't now why, can it be biome if I migrated from prettier with tailwind css plugin...
Why?

v4 beta is out
Seems like resolveConfig is no longer exported causing eslint to crash during loading:
[Error - 4:03:00 PM] Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './resolveConfig' is not defined by "exports" in [redacted]/node_modules/tailwindcss/package.json
    at exportsNotFound (node:internal/modules/esm/resolve:304:10)
    at packageExportsResolve (node:internal/modules/esm/resolve:651:9)
    at resolveExports (node:internal/modules/cjs/loader:592:36)
    at Module._findPath (node:internal/modules/cjs/loader:669:31)
    at Module._resolveFilename (node:internal/modules/cjs/loader:1131:27)
    at Module._load (node:internal/modules/cjs/loader:986:27)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> ([redacted]/node_modules/eslint-plugin-tailwindcss/lib/util/customConfig.js:5:23)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)

Really keen to see Tailwind v4 support <3


change Azure CLI command for status check of registration

Missing g in azure CLI

@ikshitiz : Thanks for your contribution! The author(s) have been notified to review your proposed change.

This has been fixed in a separate PR. The doc has been updated for GA readiness as well. You can close this PR


httproute: use lists insteads of maps for HTTPRequestHeaderFilter

What type of PR is this?

/kind api-change
/kind cleanup
What this PR does / why we need it:
This matches the API conventions that prefer lists of named subobjects over maps.
Which issue(s) this PR fixes:

Fixes #658
See #657
Does this PR introduce a user-facing change?:

Headers in Request modifier filter are now represented with lists of named subobjects instead of maps.

/cc @robscott

@jpeach @robscott I had made a copy pasta error. PTAL.

One thing to note:
I tried to do struct embedding here but kubebuilder CRD generator didn't like that. I know upstream doesn't recommend embedding unless it absolutely makes sense, which I think is the case here. Should we report this problem to kubebuilder community or is this expected?

I tried to do struct embedding here but kubebuilder CRD generator didn't like that. I know upstream doesn't recommend embedding unless it absolutely makes sense, which I think is the case here. Should we report this problem to kubebuilder community or is this expected?

I don't have a strong opinion on struct embedding here, but don't think it needs to block this PR getting in. What happened to the CRD, was the yaml just inaccurate/unexpected? If so, I think it's worth filing a bug on controller-tools repo with a reference to the command we're using to generate CRDs.

@hbagdi This looks like it's close, let me know when it's ready for me to take another look.

@robscott updated. PTAL.

Thanks @hbagdi! This LGTM but looks like it needs another make generate run.

Thanks!
/lgtm


Move studio docs (cn)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.4 out of 8 committers have signed the CLA.:white_check_mark: dutor:white_check_mark: whitewum:white_check_mark: Amber1990Zhang:white_check_mark: oldLady344:x: wilsonyou:x: NicolaCage:x: nianiaJR:x: hetao92You have signed the CLA already but the status is still pending? Let us recheck it.


Masonry measurementStore example?

I've googled and searched this repo but can't find a concrete example usage for Masonry's measurementStore prop.
    <Masonry
      items={items}
      comp={MasonryCard}
      scrollContainer={() => container as HTMLElement}
      minCols={1}
      virtualize
      layout="MasonryDefaultLayout"
      columnWidth={columnW}
      gutterWidth={gutterWidth || 20}
      virtualBoundsBottom={400}
      virtualBoundsTop={400}
      loadItems={itemLoader}
      measurementStore={} // How to do this?
    />

The docs say to implement it using Masonry.createMeasurementStore(), but while I can see that there's a constructor in the class I can't see how to implement this when applied to props.
Thanks in advance!

@liuyenwei @btraut Could either of you answer this one? I can describe what I'm reading in the code for our various uses of Masonry, but I'll probably miss the subtleties of the intent behind the measurementStore. (I'm also adding a ticket to update our Masonry docs)

@bjfresh sorry for the lack of documentation on this feature.
The usage is actually pretty straightforward:
const measurementStore = Masonry.createMeasurementStore();

const MyComponent = () => (
  <Masonry
    measurementStore={measurementStore}
  />
);

The most important point with createMeasurementStore is that it needs to be declared in module scope (vs inside your component).  The reason for this is because the measurement store is meant to cache layout information so we want it to persist between mounting/unmounting of a component.

@bjfresh sorry for the lack of documentation on this feature.
The usage is actually pretty straightforward:
const measurementStore = Masonry.createMeasurementStore();

const MyComponent = () => (
  <Masonry
    measurementStore={measurementStore}
  />
);

The most important point with createMeasurementStore is that it needs to be declared in module scope (vs inside your component). The reason for this is because the measurement store is meant to cache layout information so we want it to persist between mounting/unmounting of a component.

Sorry, I cannot find it in Masonary:
Property 'createMeasurementStore' does not exist on type 'typeof Masonry'.ts(2339)
version "45.8.7"

@liran319 It looks like you're perhaps using Gestalt with TypeScript? Unfortunately we currently only offer Flow types…is the method actually not available, or is it just that TS doesn't recognize it?

@liran319 It looks like you're perhaps using Gestalt with TypeScript? Unfortunately we currently only offer Flow types…is the method actually not available, or is it just that TS doesn't recognize it?

I think you are right, since I just modify to (Masonry as any).createMeasurementStore(), it works.
Thanks!


Daily gapplication-service crashes

Describe the bug
Using:  walker -m clipboard
Module(s) not found
.⏎
Version
0.8.8-git and 0.8.7
Minimal Config
Even with default config.
Desktop

Arch
Hyprland

I'm not sure what happened, but it is now working again.   Keeping this open for a few days.

Okay I figured what happened there.
Basically the clipboard module is only available when the background service is running.
If you run walker -m clipboard without having launcher --gapplication-service before end you will get "Module(s) not found"
Probably Walker crashed at some point and so I could not access my clipboard anymore.
I'm not sure what would be the best UX for this case, but I don't like it too much that is fails silently when using keybinds.  It was rather frustrating and it's not very good experience if you need to hop in a terminal and type the command by hand to check what error there is.
Maybe this is a good opportunity to implement basic error/status messages in the UI itself.
So if one was to call "walker -m clipboard" without the service running it would show a message:  "Clipboard module is unable until the walker service is running"  or something along theses lines.
Rofi is doing it , if your command fails to execute, you get a simple window with some error description and help.

You could probably also probably have the gapplication-service starts automatically if someone call one of the module that requires it to run: Clipboard and Windows (switcher)

Starting it automatically is impossible, as it's up to the user how to even start it. Some people have a systemd service, i run exec-once in Hyprland. Not sure if I'm missing something rn, but i wouldn't know how to start it.
Would by nice to know why it crashed.
I thought about utilizing notify-send to send a notification when the service crashes. I think that'd be good in general.
I agree, a more UX friendly failure UI would be good.

Hey, depends on how you launch it. Since it's just writing to stdout ... i assume you should be able to retrieve it?
How often does that happen? You'd ofc also just open up a terminal and let it run in there until it crashes.

❯ walker --gapplication-service
INFO Vulkan: Loader Message: linux_read_sorted_physical_devices: priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:      Original order: priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [0] AMD Radeon RX 7800 XT (RADV NAVI32) priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [1] AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:      Sorted order: priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [0] AMD Radeon RX 7800 XT (RADV NAVI32)   priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [1] AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO)   priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message: linux_read_sorted_physical_devices: priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:      Original order: priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [0] AMD Radeon RX 7800 XT (RADV NAVI32) priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [1] AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:      Sorted order: priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [0] AMD Radeon RX 7800 XT (RADV NAVI32)   priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:            [1] AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO)   priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message: vkCreateDevice layer callstack setup to: priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:     priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:      || priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:     priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:      || priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:     priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
INFO Vulkan: Loader Message:        Using "AMD Radeon RX 7800 XT (RADV NAVI32)" with driver: "/usr/lib/libvulkan_radeon.so" priority=6 code_file=../gtk/gdk/gdkvulkancontext.c code_line=1692 code_func=gdk_vulkan_debug_report glib_domain=Gdk
panic: runtime error: slice bounds out of range [:-8]
goroutine 23 [running]:
github.com/neurlang/wayland/wl.(*BytePool).Take(0xc000428006?, 0xfffffffffffffff8)
/home/runner/go/pkg/mod/github.com/neurlang/wayland@v0.1.36/wl/utils.go:31 +0x9c
github.com/neurlang/wayland/wl.(*Context).readEvent(0xc00023e460)
/home/runner/go/pkg/mod/github.com/neurlang/wayland@v0.1.36/wl/event.go:74 +0x370
github.com/neurlang/wayland/wl.(*Context).run(0xc00023e460, 0x0)
/home/runner/go/pkg/mod/github.com/neurlang/wayland@v0.1.36/wl/context.go:165 +0x2d
github.com/neurlang/wayland/wl.(*Context).Run(...)
/home/runner/go/pkg/mod/github.com/neurlang/wayland@v0.1.36/wl/context.go:147
github.com/abenz1267/walker/internal/modules/windows/wlr.StartWM(0x9a9be5?, 0x0?)
/home/runner/work/walker/walker/internal/modules/windows/wlr/wm.go:64 +0x17e
created by github.com/abenz1267/walker/internal/modules/windows.(*Windows).SetupData in goroutine 100
/home/runner/work/walker/walker/internal/modules/windows/windows.go:34 +0x28

Took less than an hour to get it to crash today.   Not sure what trigger the crash, I'll continue monitoring.
From what I remember from the last  hour (and what I can see was saved to the clipboard) I only used the default mode to run apps, and some copy/paste to the clipboard.

Hm, i can't really digest what's causing the runtime error in the wayland library....

Seems the lib receive a null sized event, and the allocation fails.
	// subtract 8 bytes from header
	data := bytePool.Take(int(size) - 8)

I may have to open an issue upstream.

Yeah, THAT i figured out as well, but I can't figure out why it works fine on my system. Like... i use my computer for like 52 hours per day and it never crashes.

Do you use the windows switcher much?

I believe I have found the main culprit to my crashes.
I have a custom walker module that use a bash script  and in some specific conditions the script might return nothing (ie : no columns separated by tabs)   Walker does not seems to validate the string that it receive from the script, so it crashs.
I get the following crash:

panic: runtime error: index out of range [1] with length 1
goroutine 12 [running]:
github.com/abenz1267/walker/internal/modules.Plugin.Entries({{{0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}, 0x1, {0xc0002fc1a0, 0x7}, ...}, ...}, ...}, ...)
/home/runner/work/walker/walker/internal/modules/plugins.go:120 +0xaea
github.com/abenz1267/walker/internal/ui.processAsync.func3({0xe3a748, 0xc000316410}, 0x0?, {0x0, 0x0}, {0xe3c6a0, 0xc000002d80})
/home/runner/work/walker/walker/internal/ui/interactions.go:733 +0xe4
created by github.com/abenz1267/walker/internal/ui.processAsync in goroutine 90
/home/runner/work/walker/walker/internal/ui/interactions.go:726 +0x93f

Step to reproduce.
1- Add this module to walker's config.json
    {
      "name": "testmod",
      "placeholder": "Hello World",
      "keep_sort": true,
      "prefix": "~",
      "terminal": true,
      "refresh": true,
      "label_column": 1,
      "result_column": 2,
      "separator": "\t",
      "src_once": "echo -e 'foo'",
      "cmd": "something"
    }

2- Start the service in a terminal:

walker --gapplication-service

3- In another terminal start the module, and it will crash instantly.

walker -m testmod

It would be nice to improve overall stability by bullet proofing the code from invalid input.

Hm, yeah.. that looks odd. But isn't that just a mis-configuration? What would you like the output to be in that case, since an explicit label column and result column is specified. just foo?

Of course this can be fixed in the script itself (which I did for now)
But I strongly believe Walker should avoid crashing at any cost.,  as it is running a service, and it holds the clipboard. It's kind of frustrating to realize a bunch of stuff you copied to the clipboard is actually gone, and that the program has crashed 15 minutes ago.
I would just add some validation code to the input to avoid crashing and would fallback to display the usual No Result.
Does that make sense to you ?

Yes, it does. I just wasn't sure what a user would expect for "broken" entries. I'll just discard them then.

Should be fixed.

Thanks!  Closing this one. Will keep an eye on stability on my setup.


Fix AggregateField behavior

Fix AggregateField behavior.
Some cleanups on Archivable behavior (copyright info and unused namespaces).

LGTM 👍


How to cleanly handle overlapping memory regions?

So, I've been working on implementing RL78 in Sleigh, and I've hit an edge case I can't seem to find a way around that really seems to be messing with the decompiler.
Basically, in RL78, the memory space is 20 bits, but the stack pointer is 16 bits. To handle this, the architecture engineers restricted the stack space to the range from 0xF0000 to 0xFFEFF (0xFFF00 - 0xFFFFF is the register page). I can work around this one of 2 ways, both of which have issues:

writing macros that always add a zero-extended SP into a constant of 0xF0000 and using that address for pop/push, but this screws up the decompiler because the stack accesses don't line up with where it thinks the stack is, as far as I can tell.
defining the stack as its own space, which fixes the issue of bad decompiling, but fails to take into account the fact that this is a unified memory space where the stack can be written to via other methods.

The preferred way I'd fix this would be one of a few methods -- first, just use one address space and create some kind of virtual register of some sort that is the stack pointer in the lower 16 bits and a constant in the upper 4, so that way I could set   <stackpointer register="virtualSP" space="ram"/>
in the .pspec file and have it just play nice.
--second, do some kind of nested memory mapping, such that the stack region is functionally just an offset into the ram space. bit/byte mapping seems like it would maybe do this, except that it doesnt seem to actually allow the kind of clean handling im hoping for.
Is there any way to accomplish this kind of nasty mapping and have the decompiler understand that the stackpointer has an implicit offset? Or am I just SOL on this one?

There is a third method that you can use that has been done in other processor modules like the 16-bit x86 where you essentially have two parts to the address access.  If both parts aren't known, then the decompiler will use the value of the SP as an offset.  The segment() user op is used along with some pcode attachment in the .cspec.
I would try your second route first.  I think this will work but there might be issues I'm not thinking about.  Define the byte mapped stack block, but don't actually initialize the bytes in the .pspec file.  Create a byte mapped memory block in the stack address space on top of the ram address space.  I think this should work even though the address spaces are slightly different sizes.  The stack address space would be size 2, and the main memory size 3.
There can be side-effects from either method, and sometimes they aren't apparent until you try it.

So, when you say the second, do you mean the second ifeal, with the nested memory mapping? Because if so, how would I go about that? Byte-mapping creates 2 separate memory spaces, and ideally I'd like it to be clear that this is 1 whole

Sorry accidentally closed issue while typing out response. Reopening while I finish typing this out.

So, when you say the second, do you mean the second ideal, with the nested memory mapping? Because if so, how would I go about that? Byte-mapping creates 2 separate memory spaces, and ideally I'd like it to be clear that this is 1 united memory space. Byte mapping a stack-specific region over ram is what I'm doing now, and it's definitely messier than I'd like. Would there be a way to make a single big memory space that represents all of memory, byte-map the different regions into that memory space, and then hide the component segments so that it appears as one unified space? If not, and I were willing to modify the codebase and go about submitting a pull request, how would I start?
Also, could you maybe give a bit more detail about how the x86 solution you mentioned works? Any idea which files in the x86 sleigh rats nest I should look at? I'd like to at least give it a shot, see what kind of results it yields.

(for what it's worth, I've been pushing my (extremely messy, sorry, still getting the hang of this) sleigh code to github.com/hedgeberg/rl78_sleigh if you have thoughts on what to change/where. The ISA programmer's manual for the arch is in the docs folder, if the behavior here is unclear)

Hey, seems like you guys are super busy and I've currently got my sleigh stuff working well enough fo the time being via byte mapping, so I'm going to go ahead and close this to get it out of your queue. Thanks for the help @emteere.


Created two tools for updating recommended change markdown files

RecommendedChangeMarkDownReader: Read recommended change markdown files to an excel file for each editing
RecommendedChangeMDWriter: Create markdown files from an excel file. The markdown files are organized by namespaces and unique recommended changes.

Any reason you're not just making a new repo for these?

Any reason you're not just making a new repo for these?

It may be a better option. I thought of checking into this repo is because the md files are in the same repo. However, they don't have to be in the same repo, separating these tools from product repo may be better choose.


MSID: 101992 Version: 1 DOI: 10.1101/2<IP_ADDRESS>0060

MSID: 101992
Version: 1
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>0060
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=101992
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>0060

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/101992v1
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>0060
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/101992/v1/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!

Reviews posted to the v1, but we should be using the v2.
I've moved them over. Now waiting for the DocMap to update/workflow to re-run.


Clarification on portability of RX.NET?

I'm quite confused as to what the status of Rx.Net is with regards to its portability. Specifically, .Net 5.0 is being targeted at both OSX and Linux, so where does that leave Rx.Net? I see that Rx-Main has a dependency on PlatformServices, which scant information (found via Google) seems to suggest is not particularly portable. I'm experimenting with building a portable game engine using .Net and I'm trying to make decisions that won't negatively impact my ability to port the core components to other platforms. If I take a dependency on Rx-Main now, am I going to be kicking myself a year down the road and having to refactor the whole thing out in favour of some other event streaming library?

Rx is portable and usable on Mono - I don't have personal experience with this, but I know that ReactiveUI which uses Rx runs on the Xamarin platform on iOS and Android.
PlatformServices is an abstraction designed to take advantage of platform specific capabilities where they exist. There's a fairly good explanation of how it works in this video - note that although this is from August 2012, the concept remains unchanged.

Correct. The PlatformServices assembly is intentionally not portable and contains all the platform-specific capabilities that can be used to "light up" the portable functionality in Interfaces, Core, Linq, and Providers assemblies. You can safely take a bet on the portability of Rx. Let us know if you run into any issue.

Ok, thanks!


Consolidate Simulation Parameter Metadata into Core Library

To ensure that the necessary metadata for each parameter (such as range, human-readable labels, etc.) is available to the UI, we should centralize this data within the core library. This metadata is also crucial for parsing and model validation. By consolidating these definitions, we can avoid duplication, thereby reducing the risk of inconsistencies and outdated information.

A Database is a possible solution, but it should be local for each user. We would use SQLite.
As every Element has different Parameters this could be a Problem in a strict database model.
We could save whole Beamlines without xml or toml etc.


SPIKE: Learn about CLBC Privacy info request, create tickets

Describe the issue
Audra and Gen need a hand responding to some CLBC questions.  Gary is still in training but it would be valuable to have a discussion to capture the needs for this stream of work and create tickets.
Additional context
Add any other context, attachments or screenshots
Definition of done

Meeting with Audra to outline the context and desired outcomes
Create tickets for next sprint that are broken up into achievable units of work in one sprint, have a clear DoD with well defined deliverables (will make a Word doc on Teams, ship to Audra), explain any needed QA steps with Audra/Gen

Done because tickets exist now


Better error handling when an extension doesn't exist

If the shared preload library for "pg_tle" is loaded but the "pg_tle" extension is not activated and a user tries to install an extension that does not exist, pg_tle would give an error of the form "cannot find pgtle schema". However, while true, this is not a helpful error message.
The issue was due to the ProcessUtility hook firing even though the "pg_tle" extension itself was not active. To avoid this, we add a check to the PU hook to only run its custom logic when we know that "pg_tle" is installed.
fixes #187

Some notes to consider:

Move the load checks to a function to better encapsulate.
Consider moving the standard PU execution portion to its own call (though note the compatibility layer in place)
Add tests 😄


Refactor color classes

render/block/BlockColorMapper -> color/block/BlockColorProvider
render/block/BlockColorMap -> color/block/BlockColors

getRenderColor -> getColor (returns material color if no multiplier present)
getColorMultiplier (returns -1 if no multiplier present)

render/item/ItemColorMapper -> color/item/ItemColorProvider
render/item/ItemColorMap -> color/item/ItemColors

getRenderColor -> getColorMultiplier (returns -1 if no multiplier present)

render/block/BiomeColors -> color/world/BiomeColors

$Proxy -> $Provider
colorAt -> getColor
foliageColorAt -> getFoliageColor
grassColorAt -> getGrassColor
waterColorAt -> getWaterColor

render/block/FoliageColorHandler -> color/world/FoliageColors
render/block/GrassColorHandler -> color/world/GrassColors

You keep asking on Discord for a review, but I don't have any context on this and I do not know why the renames were done and I cannot judge whether this might be a good or a bad change.

the new names seem fine to me, but ill leave this open for review to someone else who knows this area a bit better.


 You're the lucky last commiter :)

@jterry75 - 1975 was a good year. I started working on this ridiculous project, so please stop me before it's too late: https://github.com/EliteLoser/misc/tree/master/PowerShell/STDockerPS


feat(admin): add poll page

add poll management system.

@SUNNYKUMAR1232 it would be great if you add add screen shorts of the pages.


Run perf tests under federated auth

Run the tests under the same federated auth used to deploy the tests and setup the variables needed to configure AzurePipelineCredential.

/azp run azure-sdk-tools - sync - eng-common

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow


[WIP]Introduce eBPF prog coverage

What type of PR is this?

/kind enhancement
What this PR does / why we need it:
Introduce eBPF prog coverage
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

Introduce a new flag `enable-bpf-coverage` to generate eBPF code coverage after Kmesh is exited

How to generate coverage

Start Kmesh with --enable-bpf-coverage=true
Copy all codes in bpf/ to Kmesh-pod:/kmesh
Run test cases
Get yourself into Kmesh-pod, then stop Kmesh
The coverage HTML is at /kmesh/sockops-cov.html

How to generate coverage

Start Kmesh with --enable-bpf-coverage=true
Copy all codes in bpf/ to Kmesh-pod:/kmesh
Run test cases
Get yourself into Kmesh-pod, then stop Kmesh
The coverage HTML is at Kmesh-pod:/kmesh/sockops-cov.html

can you introduct it at Thursday's community meeting?

How to generate coverage

Start Kmesh with --enable-bpf-coverage=true
Copy all codes in bpf/ to Kmesh-pod:/kmesh
Run test cases
Get yourself into Kmesh-pod, then stop Kmesh
The coverage HTML is at Kmesh-pod:/kmesh/sockops-cov.html

can you introduct it at Thursday's community meeting?

Sure

/retest


Android build failing

Not able to compile the code after including the cluster code
Error: Command failed: gradlew.bat app:installDebug -PreactNativeDevServerPort=8081
FAILURE: Build failed with an exception.

What went wrong:

A problem occurred configuring project ':react-native-clusterer'.

com.android.builder.errors.EvalIssueException: NDK at C:\Users\dell\AppData\Local\Android\Sdk\ndk\21.4.7075529 did not have a source.properties file

I would need more details or preferably a repository containing this issue to reproduce it.


PrevMonth & NextMonth don't work correctly after upgrade to 4.1.1

After recently the updated to 4.1.1 from 4.1.0, the PrevMonth & NextMonth arrows in DayPicker don't work as expected. Tested in Chrome (54.0.2840.98 64-bit) & Safari.
The NextMonth doesn't work at all (month won't change), whereas PrevMonth will cause 2 months overlay at first click and then stop working as well, like below.

However, in Chrome, if click (i guess because of the focus effect) within the days area first, they can work correctly.
In Safari, even after click a day, the Prev & Next can only work once and initial month will never appear again (the element is still in DOM, but data-visible won't be set to true), for example, start with Dec, u can at most go next to Jan and then Prev will lead directly to Nov, vice versa.

Any known changes in 4.1.1 will cause above situation?

Yikes! This is true in the storybook, but doesn't seem to affect the DRP/SDP. I will investigate ASAP.
It seems likely to be related to https://github.com/airbnb/react-dates/pull/196.

Got a PR up! 👍 Thanks for flagging!

Okay, should be fixed in v4.1.2! :)


feat: TimePicker

Notes
Add a TimePicker component which wraps Material UI's KeyboardTimePicker.
Integrate with FormRenderer for the time-picker component type (componentTypes.TIME_PICKER).
Times are represented as Date objects when using the TimePicker component directly.
In FormRenderer, times are represented as ISO 8601 strings for simpler payloads sent to services.

Testing
npm run test
Inspected storybook and styleguide.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

:tada: This PR is included in version 1.0.22 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Added time module and TimeNow Node

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 63.15789% with 7 lines in your changes missing coverage. Please review.

Project coverage is 33.20%. Comparing base (0fef933) to head (8d1b528).

Files with missing lines
Patch %
Lines

ros_bt_py/ros_bt_py/ros_nodes/time.py
61.11%
7 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##              dev     #111       +/-   ##
===========================================
- Coverage   49.40%   33.20%   -16.21%
===========================================
  Files          58       39       -19
  Lines        7681     5824     -1857
  Branches     1658     1296      -362
===========================================
- Hits         3795     1934     -1861
- Misses       3822     3833       +11
+ Partials       64       57        -7

Flag
Coverage Δ

?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update WorkManager to 1.0.0-alpha08

@tiembo I have tried the upgrade locally, it works fine, whatz wrong on CI?

@XinyueZ I think it was a temporary CI issue


Request: use negation, exact sigils after search text

[x] I have read through the manual page (man fzf)
[x] I have the latest version of fzf
[x] I have searched through the existing issues

Info

OS

[x] Linux
[ ] Mac OS X
[ ] Windows
[ ] Etc.

Shell

[x] bash
[ ] zsh
[ ] fish

Problem / Steps to reproduce
say you want to search for 'very_long_word', fzf shows a lot matches but you realize that it's an exact match you want, you now have to scroll back, add a quote, scroll to the end to add further elements to the search. This happens daily to me, often enough so I wrote this request.
having the possibility to add the quote or the exclamation mark after 'very_long_word' would be great.

you now have to scroll back, add a quote, scroll to the end

I'm pretty used to pressing ALT-B ' ALT-F in that case. It won't take more than a second once you get used to it.
Allowing ' or ! at the end of a word is basically changing the search syntax of fzf thus breaking backward compatibility,
suddenly disallowing users to search foo!, so I'm reluctant to make this change.
Having said that, the closest thing you can do at the moment is something like this.
fzf --bind "':backward-word+put+forward-word"


"$lt" isnt working.

i am using following selector. ["datetime" has a long date value i.e [1434054660000] , "patientID" has a int value i.e.[1] ]

selector: {
  datetime: { "$lt": datetime },
  patientID: { "$eq": parseInt(patientID) } ,
  glucoseType: { "$eq": 'fasting' }}

But for some reason it does not seem to work.
Results are empty Array[0], even when i have records matching the query.
No Error is looged in colsole.
Following is the Input for Query Plan creation that is logged :
Object { selector: Object, sort: Array[1], limit: 1}
    limit: 1
    selector: Object
         datetime: Object
               $lt:<PHONE_NUMBER>000
               __proto__: Object
         glucoseType: Object
               $eq: "fasting"
               __proto__: Object
         patientID: Object
               $eq: 1
               __proto__: Object
        __proto__: Object
   sort: Array[1]
        0: Object
            datetime: "desc"
            __proto__: Object
            length: 1
        __proto__: Array[0]
   __proto__: Object

Following is the Query Plan that is printed in debug log :
Object {queryOpts: Object, index: Object, inMemoryFields: Array[2]}
inMemoryFields: Array[2]
      0: "glucoseType"
      1: "patientID"
      length: 2
      __proto__: Array[0]
index: Object
    ddoc: "_design/idx-20dd3f8cc9deca1f70feb4d70383e7a3"
    def: Object
           name: "idx-20dd3f8cc9deca1f70feb4d70383e7a3"
           type: "json"
           __proto__: Object
queryOpts: Object
     endkey: Array[3]
           0:<PHONE_NUMBER>000
           1: "fasting"
           2: 1
           length: 3
           __proto__: Array[0]
           inclusive_end: false
     startkey: Array[3]
          0: null
          1: "fasting"
          2: 1
          length: 3
          __proto__: Array[0]
     __proto__: Object
__proto__: Object

Suspecting this is a duplicate of #38.

What does your index look like? Need more information in order to reproduce this.

This issue was moved to pouchdb/pouchdb#6288


Live-update mobile without going through app stores

One of the most amazing features of RN is the ability to "live edit" without going through the app store.  If we can do that, that would be truly amazing as it means we only need to go to the app store when we are doing a "major new feature" (though even that is just advisory, there's no actual technical requirement for that).

I think now is as good a time as any to reopen this investigation

I'm not sure I'll ever get around to this. I'm going to unassign myself and let it decay unless someone else wants to work on it.

@quinthar, this Monthly task hasn't been acted upon in 6 weeks; closing.
If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.


[SPARK-41572][SQL] Assign name to _LEGACY_ERROR_TEMP_2149

What changes were proposed in this pull request?
This PR proposes to assign name to _LEGACY_ERROR_TEMP_2149, "MALFORMED_CSV_RECORD".
Why are the changes needed?

We should assign proper name to LEGACY_ERROR_TEMP*
Does this PR introduce any user-facing change?

No
How was this patch tested?

Fix UT & run ./build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite*

thanks, merging to master!


xh: 0.21.0 -> 0.22.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/ducaale/xh/releases.
meta.description for xh is: Friendly and fast tool for sending HTTP requests
meta.homepage for xh is: https://github.com/ducaale/xh
meta.changelog for xh is: https://github.com/ducaale/xh/blob/v0.22.0/CHANGELOG.md
Updates performed

Rust version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.22.0 with grep in /nix/store/z0hpqyh55psw4jpmbfg8j9d5zv6zbb4s-xh-0.22.0
found 0.22.0 in filename of file in /nix/store/z0hpqyh55psw4jpmbfg8j9d5zv6zbb4s-xh-0.22.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
xh

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/z0hpqyh55psw4jpmbfg8j9d5zv6zbb4s-xh-0.22.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A xh https://github.com/r-ryantm/nixpkgs/archive/d3ea189a5aa8e0959757555a9368535a286eebef.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/d3ea189a5aa8e0959757555a9368535a286eebef#xh

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/z0hpqyh55psw4jpmbfg8j9d5zv6zbb4s-xh-0.22.0
ls -la /nix/store/z0hpqyh55psw4jpmbfg8j9d5zv6zbb4s-xh-0.22.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

xh

Maintainer pings
cc @figsoda @bhankas for testing.

@NixOS/nixpkgs-merge-bot merge


Formal verification pmp

The verification time is pretty high for that one (700s on my machine), we'll have to see if we can find ways to make it more efficient.


fix: update gobin

What this PR does / why we need it: Title says, it does.

JIRA ID: XX-XX

Notes for your reviewer:

:tada: This PR is included in version 1.33.2 :tada:
The release is available on:

GitHub release
CircleCI Orb

Your semantic-release bot :package::rocket:


TST: use pytest

Make tests pytest based. Also use pytest-xdist for parallel testing (~1.5x speedup on CI, but {n_cores} speedup on local HW).
Also added poetry as a package/dep manager.
I was having reproducibility issues with pulling the mnist data, I switched to Google/Keras' mirror of it and seemed to get better results. I can probably undo all of those changes if I test a bit more with the original implementation @geohot, let me know.

So this is a ton of lines and hard to merge. I see multiple distinct things here:

Refactor/change the mnist data loader (don't like all the globals, is filelock an added depth now?)
Add poetry. I don't think this should have a package manager (it's tiny!) but I'd be fine with a requirements.txt with very normal packages in it.
Refactor mnist -> test_mnist (completely impossible diff to read)

Can you resubmit these things as separate PRs?

Can you resubmit these things as separate PRs?

Totally, I put a minimal version of this in #2

Refactor/change the mnist data loader (don't like all the globals, is filelock an added depth now?)

This is only required if you want to be able to parallelize tests. Is parallelizing tests something you want? Once you have pytest, it's just 1 extra plugin. Only worth it if you think you'll have more / longer running tests.

Add poetry. I don't think this should have a package manager (it's tiny!) but I'd be fine with a requirements.txt with very normal packages in it.

Sure, I'll do a seperate PR (after #2) with just a requirements.txt


async  can use it at Typescript?

async function getStockPriceByName(name) {
return name;
}
error TS2468: Cannot find global value 'Promise'.
test.ts(1,16): error TS2705: An async function or method in ES5/
ES3 requires the 'Promise' constructor.  Make sure you have a de
claration for the 'Promise' constructor or include 'ES2015' in y
our --lib option.
{
"compilerOptions": {
"target": "es5",
"experimentalDecorators": true,
"lib": [
"dom",
"es5",
"es6"
]
}
}

You have to specify your tsconfig.json for tsc so that TS can pick up your configs.
tsc -p path/to/your/tsconfig.json

tsc test.ts is considered using default settings.

tsc -p ./tsconfig.json
still  erro

solved by
{
"compilerOptions": {
"target": "es5",
"experimentalDecorators": true,
"lib": [
"dom",
"es5",
"es6"
]
},
"files": [
    "test.ts"
]

}

{
"compilerOptions": {
"target": "es5",
"experimentalDecorators": true,
"lib": [
"dom",
"es5",
"es6"
]
},
"files": [
    "test.ts"
]

}
solved


My playgroud can't find the TJBot

The instruction says "please make sure the tjbot-daemon process is running on your TJBot". But how do I check whether it is running?

I am having the same issue and would appreciate further guidance.


README.md: made the description of .one() clearer

I was confused as to what "Remember you can also bind once" meant until I tried it.  So I've made a minor adjustment to explain it more clearly.

Thx!


Add other vegetables as a source of starch?

http://minecraft.curseforge.com/projects/the-vegan-option?comment=4

I'll have to look into which other vegetables would work for obtaining starch. I'm pretty sure potato starch is somewhat unique in terms of its use as an egg replacer, and I'm not sure which starches are useful for bioplastic. I used potato starch specifically because I know it's possible to create your own potato starch-based bioplastic without any fancy techniques.

What about Corn Starch, if Harvestcraft or another modding adding corn is installed?
https://en.wikipedia.org/wiki/Corn_starch

Yeah, corn seems reasonable. Will add that.


Javascript error in version '0.12.0', some pages cannot be open

Javascript error in version '0.12.0'.
Uncaught Error: Syntax error, unrecognized expression: a.src_link[href=#4eb942230860ef99bbe56de21801de7c3054304f]
    at Function.w.error (<anonymous>:1:28701)
    at w.tokenize (<anonymous>:2:1891)
    at w.select (<anonymous>:2:2156)
    at Function.w [as find] (<anonymous>:1:17673)
    at Te.fn.init.find (<anonymous>:2:4445)
    at new Te.fn.init (<anonymous>:2:5054)
    at Te (<anonymous>:1:14075)
    at HTMLDocument.<anonymous> (<anonymous>:7:5122)
    at t (<anonymous>:2:9165)
    at a (<anonymous>:2:9471)

I ran unit test with simplecov-html 0.11.0, no error occurred.
But when I upgrade simplecov-html into version 0.12.0, when I open some of the code coverage items(including nearly all the controllers and more than half of the helpers, but the coverage percentage value is correct), this error is thrown.
My ruby version is 2.6.5,  rails version is <IP_ADDRESS>, simplecov version is 0.18.2.

:wave:
Hi @HansBug thanks for opening an issue!
Could you please include code that throws the error? It seems that the code highlighter is erroring out at some point. Without the code to reproduce this error, it's nigh impossible for us to fix :)
Cheers,
Tobi

Emmm, I'm really sorry, the code on our side is not convenient to disclose directly for some reasons (large scale, and it is temporarily impossible to determine what this error is related to).
I will try the different versions (0.11.0 and 0.12.0) of the code to reproduce this error. May be wait for some moments.

@HansBug yeah, I can see how that's a problem. Would be mightily helpful though. It seems the last release also introduced unicode character errors (see #91 ) - so maybe it's related?
Thanks for your report and your help! :green_heart:

Emm, maybe it's different. In that case the page can still be opened properly, only the non-ascii characters cannot be displayed. But in my case, the page that shows which lines are covered and which lines not cannot be displayed (the javascript error above occurs at this time) when I click the .rb file's path.

yeah I just thought it might be related aka through the unicode error weird characters are inserted which makes the parser crash :man_shrugging:
I'm just saying, as some files work I think it has to be related to something odd happening in the code.
Highlighter was one of the view packages I did not upgrade though so it might also be that in some path it tries to access some jQuery function that isn't there anymore. Maybe updating it would solve it, but I'm honestly unsure what highlighter package it is :man_shrugging:

Same issue on my side. I have the issue on a really simple stupid file:
class Brand < ActiveRecord::Base
end

Steps to recreate using a public repo:
% ruby --version
ruby 2.5.7p206 (2019-10-01 revision 67816) [x86_64-darwin19]
% git clone https://github.com/thoughtbot/factory_bot.git
% cd factory_bot
% bundle install
% bundle update simplecov simplecov-html
% bundle exec rspec
# open ./coverage/index.html
# Click "Next" in the pagination
# Try to open any file on that page

I tested this in Chrome and Firefox using both "file:///" urls and loaded through a webserver and anything outside the first 10 results won't load.

Related https://github.com/colszowka/simplecov/issues/867

Thanks a bunch this will be helpful in fixing/debugging this :)

Yeah the more I see it now it's that just everything on the second page doesn't load to which #93 is a kinda great solution :angel:

@HansBug can you confirm that this only happens on "not the first page" ? The error I get is different and the file doesn't open at all. Just wanna find out if this is "the same" or another error.

The error referenced in this issue
Error: Syntax error, unrecognized expression: a.src_link[href=#10233c3a374374beb92db32fe83f29e616c6d742]

happens in the following scenarios.

Going to page 2+ and opening a file causes the error.
Changing the show results to 25+ and opening a file causes the error.
Viewing one of the working files then refreshing the page causes the error.

@HansBug can you confirm that this only happens on "not the first page" ? The error I get is different and the file doesn't open at all. Just wanna find out if this is "the same" or another error.

Yes, exactly! I tried this case just now, all the links in first page can be displayed correctly, all the other pages can not.

Great thank you! I'll just disable pagination pagination then :angel: #93

Fixed via https://github.com/colszowka/simplecov-html/pull/96 or at least to the best of my understanding.


create minimized version

hopefully generated automatically; see how the .zip archive is created in the tldr-pages website.

On second thought, this is not really worth the effort: the whole point of this project is to keep the ruleset to a minimum while maximizing its effects. Minimizing such a short style provides very little space-saving, while vastly reducing readability.


Ring Keypad + Alarmo v1.9.0 = no talking voices

Alarmo Version
v1.9.0
HA Version
2022.5.0
Bug description
HASSIO v2022.5.0 using Ring keypad + Alarmo v1.8.6 = talking voices when (dis)arming.
Upgraded to Alarmo v1.9.0 = no voices/lights
Deleted custom_component folder, WinSCP'd Alarmo v1.8.6 and reloaded core = voices work again.
Steps to reproduce
No response
Relevant log output
I need to enable some form of debugging for this?

I don't really know how to help, this issue seems more related to the blueprint that you are using than alarmo itself.
I would suggest to just try arming/disarming alarmo manually (e.g. via alarmo-card), see if all works as you would expect.
If so, the problem is more likely caused by the blueprint, and I would suggest to ask for help in the forum thread instead.
I cannot think of any recent changes in Alarmo which could affect (break) this blueprint.
You can enable debug logging for alarmo by adding this to the configuration.yaml of HA and restarting HA:
logger:
  default: warning
  logs:
    custom_components.alarmo: debug

It wasn't very clear from your original report that the issue was specific to countdown time indication, which I definitely did change. For next time: please be more specific when reporting a bug.
This should be fixed by updating to release v1.9.1.

Thanks @nielsfaber! I'll update and give it a whirl. Your work on this is greatly appreciated!!!!


Added test which triggers asan error with text property (issue #4412)

This PR adds a test which triggers an global memory overflow
detected by asan when using text property (see issue #4412).
Valgrind can't detect such bugs.
I'd expect Travis CI to detect this bug with the asan build.
PR should not be merged until there is a fix for the bug.

Dominique wrote:

This PR adds a test which triggers an global memory overflow
detected by asan when using text property (see issue #4412).
Valgrind can't detect such bugs.
I'd expect Travis CI to detect this bug with the asan build.
PR should not be merged until there is a fix for the bug.

It indeed fails on Travis.  Not sure how to fix it, since it's not clear
what causes the problem.  When I run the commands in a debugger it
doesn't even get to the code that ASAN complains about...
--
Apparently, 1 in 5 people in the world are Chinese.  And there are 5
people in my family, so it must be one of them.  It's either my mum
or my dad.  Or my older brother Colin.  Or my younger brother
Ho-Cha-Chu.  But I think it's Colin.
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\\  an exciting new programming language -- http://www.Zimbu.org        ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

Included with 8.1.1395