[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations

This patch adds managed-memory-based aggregation to Spark SQL / DataFrames. Instead of working with Java objects, this new aggregation path uses sun.misc.Unsafe to manipulate raw memory.  This reduces the memory footprint for aggregations, resulting in fewer spills, OutOfMemoryErrors, and garbage collection pauses.  As a result, this allows for higher memory utilization.  It can also result in better cache locality since objects will be stored closer together in memory.
This feature can be eanbled by setting spark.sql.unsafe.enabled=true.  For now, this feature is only supported when codegen is enabled and only supports aggregations for which the grouping columns are primitive numeric types or strings and aggregated values are numeric.
Managing memory with sun.misc.Unsafe
This patch supports both on- and off-heap managed memory.

In on-heap mode, memory addresses are identified by the combination of a base Object and an offset within that object.
In off-heap mode, memory is addressed directly with 64-bit long addresses.

To support both modes, functions that manipulate memory accept both baseObject and baseOffset fields.  In off-heap mode, we simply pass null as baseObject.
We allocate memory in large chunks, so memory fragmentation and allocation speed are not significant bottlenecks.
By default, we use on-heap mode.  To enable off-heap mode, set spark.unsafe.offHeap=true.
Compact tuple format
This patch introduces UnsafeRow, a compact row layout.  In this format, each tuple has three parts: a null bit set, fixed length values, and variable-length values:

Rows are always 8-byte word aligned (so their sizes will always be a multiple of 8 bytes)
The bit set is used for null tracking:

Position i is set if and only if field i is null
The bit set is aligned to an 8-byte word boundary.

Every field appears as an 8-byte word in the fixed-length values part:

If a field is null, we zero out the values.
If a field is variable-length, the word stores a relative offset (w.r.t. the base of the tuple) that points to the beginning of the field's data in the variable-length part.

Each variable-length data type can have its own encoding:

For strings, the first word stores the length of the string and is followed by UTF-8 encoded bytes.  If necessary, the end of the string is padded with empty bytes in order to ensure word-alignment.

For example, a tuple that consists 3 fields of type (int, string, string), with value (null, “data”, “bricks”) would look like this:

This format allows us to compare tuples for equality by directly comparing their raw bytes.  This also enables fast hashing of tuples.
Hash map for performing aggregations
This patch introduces UnsafeFixedWidthAggregationMap, a hash map for performing aggregations where the aggregation result columns are fixed-with.  This map's keys and values are Row objects. UnsafeFixedWidthAggregationMap is implemented on top of BytesToBytesMap, an append-only map which supports byte-array keys and values.
BytesToBytesMap stores pointers to key and value tuples.  For each record with a new key, we copy the key and create the aggregation value buffer for that key and put them in a buffer. The hash table then simply stores pointers to the key and value. For each record with an existing key, we simply run the aggregation function to update the values in place.
This map is implemented using open hashing with triangular sequence probing.  Each entry stores two words in a long array: the first word stores the address of the key and the second word stores the relative offset from the key tuple to the value tuple, as well as the key's 32-bit hashcode.  By storing the full hashcode, we reduce the number of equality checks that need to be performed to handle position collisions ()since the chance of hashcode collision is much lower than position collision).
UnsafeFixedWidthAggregationMap allows regular Spark SQL Row objects to be used when probing the map.  Internally, it encodes these rows into UnsafeRow format using UnsafeRowConverter.  This conversion has a small overhead that can be eliminated in the future once we use UnsafeRows in other operators.

Test build #31059 has started for   PR 5725 at commit cde4132.

Test build #31059 has finished for   PR 5725 at commit cde4132.

This patch fails MiMa tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar
tachyon-0.6.4.jar
tachyon-client-0.6.4.jar

This patch removes the following dependencies:

tachyon-0.5.0.jar
tachyon-client-0.5.0.jar

Test build #31069 has started for   PR 5725 at commit 0925847.

Test build #31069 has finished for   PR 5725 at commit 0925847.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

Test build #31095 has started for   PR 5725 at commit a8e4a3f.

Test build #31101 has started for   PR 5725 at commit b45f070.

Test build #31095 has finished for   PR 5725 at commit a8e4a3f.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public final class MemoryManager

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch removes the following dependencies:

RoaringBitmap-0.4.5.jar
activation-1.1.jar
akka-actor_2.10-2.3.4-spark.jar
akka-remote_2.10-2.3.4-spark.jar
akka-slf4j_2.10-2.3.4-spark.jar
aopalliance-1.0.jar
arpack_combined_all-0.1.jar
avro-1.7.7.jar
breeze-macros_2.10-0.11.2.jar
breeze_2.10-0.11.2.jar
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-math-2.1.jar
commons-math3-3.4.1.jar
commons-net-2.2.jar
compress-lzf-1.0.0.jar
config-1.2.1.jar
core-1.1.2.jar
curator-client-2.4.0.jar
curator-framework-2.4.0.jar
curator-recipes-2.4.0.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
groovy-all-2.3.7.jar
guava-14.0.1.jar
guice-3.0.jar
hadoop-annotations-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-client-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-app-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-mapreduce-client-jobclient-2.2.0.jar
hadoop-mapreduce-client-shuffle-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-client-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
hadoop-yarn-server-common-2.2.0.jar
ivy-2.4.0.jar
jackson-annotations-2.4.0.jar
jackson-core-2.4.4.jar
jackson-core-asl-1.8.8.jar
jackson-databind-2.4.4.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-module-scala_2.10-2.4.4.jar
jackson-xc-1.8.8.jar
jansi-1.4.jar
javax.inject-1.jar
javax.servlet-3.0.0.v201112011016.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.10.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jline-2.10.4.jar
jodd-core-3.6.3.jar
json4s-ast_2.10-3.2.10.jar
json4s-core_2.10-3.2.10.jar
json4s-jackson_2.10-3.2.10.jar
jsr305-1.3.9.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.10.jar
kryo-2.21.jar
log4j-1.2.17.jar
lz4-1.2.0.jar
management-api-3.0.0-b012.jar
mesos-0.21.0-shaded-protobuf.jar
metrics-core-3.1.0.jar
metrics-graphite-3.1.0.jar
metrics-json-3.1.0.jar
metrics-jvm-3.1.0.jar
minlog-1.2.jar
netty-3.8.0.Final.jar
netty-all-4.0.23.Final.jar
objenesis-1.2.jar
opencsv-2.3.jar
oro-2.0.8.jar
paranamer-2.6.jar
parquet-column-1.6.0rc3.jar
parquet-common-1.6.0rc3.jar
parquet-encoding-1.6.0rc3.jar
parquet-format-2.2.0-rc1.jar
parquet-generator-1.6.0rc3.jar
parquet-hadoop-1.6.0rc3.jar
parquet-jackson-1.6.0rc3.jar
protobuf-java-2.4.1.jar
protobuf-java-2.5.0-spark.jar
py4j-<IP_ADDRESS>.jar
pyrolite-2.0.1.jar
quasiquotes_2.10-2.0.1.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.10.4.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar
scalap-2.10.4.jar
scalatest_2.10-2.2.1.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
snappy-java-<IP_ADDRESS>.jar
spark-bagel_2.10-1.4.0-SNAPSHOT.jar
spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
spark-core_2.10-1.4.0-SNAPSHOT.jar
spark-graphx_2.10-1.4.0-SNAPSHOT.jar
spark-launcher_2.10-1.4.0-SNAPSHOT.jar
spark-mllib_2.10-1.4.0-SNAPSHOT.jar
spark-network-common_2.10-1.4.0-SNAPSHOT.jar
spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
spark-repl_2.10-1.4.0-SNAPSHOT.jar
spark-sql_2.10-1.4.0-SNAPSHOT.jar
spark-streaming_2.10-1.4.0-SNAPSHOT.jar
spire-macros_2.10-0.7.4.jar
spire_2.10-0.7.4.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
tachyon-0.6.4.jar
tachyon-client-0.6.4.jar
uncommons-maths-1.2.2a.jar
unused-1.0.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.5.jar

Test build #31101 has finished for   PR 5725 at commit b45f070.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public final class MemoryManager

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch removes the following dependencies:

RoaringBitmap-0.4.5.jar
activation-1.1.jar
akka-actor_2.10-2.3.4-spark.jar
akka-remote_2.10-2.3.4-spark.jar
akka-slf4j_2.10-2.3.4-spark.jar
aopalliance-1.0.jar
arpack_combined_all-0.1.jar
avro-1.7.7.jar
breeze-macros_2.10-0.11.2.jar
breeze_2.10-0.11.2.jar
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-math-2.1.jar
commons-math3-3.4.1.jar
commons-net-2.2.jar
compress-lzf-1.0.0.jar
config-1.2.1.jar
core-1.1.2.jar
curator-client-2.4.0.jar
curator-framework-2.4.0.jar
curator-recipes-2.4.0.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
groovy-all-2.3.7.jar
guava-14.0.1.jar
guice-3.0.jar
hadoop-annotations-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-client-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-app-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-mapreduce-client-jobclient-2.2.0.jar
hadoop-mapreduce-client-shuffle-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-client-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
hadoop-yarn-server-common-2.2.0.jar
ivy-2.4.0.jar
jackson-annotations-2.4.0.jar
jackson-core-2.4.4.jar
jackson-core-asl-1.8.8.jar
jackson-databind-2.4.4.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-module-scala_2.10-2.4.4.jar
jackson-xc-1.8.8.jar
jansi-1.4.jar
javax.inject-1.jar
javax.servlet-3.0.0.v201112011016.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.10.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jline-2.10.4.jar
jodd-core-3.6.3.jar
json4s-ast_2.10-3.2.10.jar
json4s-core_2.10-3.2.10.jar
json4s-jackson_2.10-3.2.10.jar
jsr305-1.3.9.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.10.jar
kryo-2.21.jar
log4j-1.2.17.jar
lz4-1.2.0.jar
management-api-3.0.0-b012.jar
mesos-0.21.0-shaded-protobuf.jar
metrics-core-3.1.0.jar
metrics-graphite-3.1.0.jar
metrics-json-3.1.0.jar
metrics-jvm-3.1.0.jar
minlog-1.2.jar
netty-3.8.0.Final.jar
netty-all-4.0.23.Final.jar
objenesis-1.2.jar
opencsv-2.3.jar
oro-2.0.8.jar
paranamer-2.6.jar
parquet-column-1.6.0rc3.jar
parquet-common-1.6.0rc3.jar
parquet-encoding-1.6.0rc3.jar
parquet-format-2.2.0-rc1.jar
parquet-generator-1.6.0rc3.jar
parquet-hadoop-1.6.0rc3.jar
parquet-jackson-1.6.0rc3.jar
protobuf-java-2.4.1.jar
protobuf-java-2.5.0-spark.jar
py4j-<IP_ADDRESS>.jar
pyrolite-2.0.1.jar
quasiquotes_2.10-2.0.1.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.10.4.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar
scalap-2.10.4.jar
scalatest_2.10-2.2.1.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
snappy-java-<IP_ADDRESS>.jar
spark-bagel_2.10-1.4.0-SNAPSHOT.jar
spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
spark-core_2.10-1.4.0-SNAPSHOT.jar
spark-graphx_2.10-1.4.0-SNAPSHOT.jar
spark-launcher_2.10-1.4.0-SNAPSHOT.jar
spark-mllib_2.10-1.4.0-SNAPSHOT.jar
spark-network-common_2.10-1.4.0-SNAPSHOT.jar
spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
spark-repl_2.10-1.4.0-SNAPSHOT.jar
spark-sql_2.10-1.4.0-SNAPSHOT.jar
spark-streaming_2.10-1.4.0-SNAPSHOT.jar
spire-macros_2.10-0.7.4.jar
spire_2.10-0.7.4.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
tachyon-0.6.4.jar
tachyon-client-0.6.4.jar
uncommons-maths-1.2.2a.jar
unused-1.0.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.5.jar

Test build #31113 has started for   PR 5725 at commit 162caf7.

Test build #31113 has finished for   PR 5725 at commit 162caf7.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public final class MemoryManager

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

I went through and left some comments. Kudos - this is super well done!

When running with off-heap, one concern is proper cleanup of managed memory when tasks fail or are cancelled.  Given that we currently do not share managed memory between tasks running on the same executor, I'm thinking that we should probably move MemoryManager from SparkEnv to TaskContext.  This would let us perform leak-detection when tasks complete; it also simplifies concurrency issues (no contention for the MemoryManager's lock when allocating / freeing blocks, provided that tasks are single-threaded) and reduces the chance that we'll exhaust our page table limit, since each task will have its own independent table.

I think good memory allocator implementations usually have a thread local cache to reduce contention. We can do the same thing too, can't we? The problem with only allocating at the TaskContext level is that you then lack global coordination, and as a result, each task is bounded by total_mem/num_tasks memory, rather than a more dynamic allocation.
Maybe a good way to do this is to have the global memory allocator in SparkEnv, and TaskContext tracks the individual pages used by each task, so it can free it.

update: thought about it more. I don't think we need to worry about contention at the moment, since our allocation unit is pretty large.

Merged build triggered.

Merged build started.

Test build #31173 has started for   PR 5725 at commit a19e066.

Merged build triggered.

Merged build started.

Test build #31185 has started for   PR 5725 at commit 70a39e4.

Test build #31173 timed out     for PR 5725 at commit a19e066     after a configured wait of 150m.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31173/
Test FAILed.

Test build #31185 has finished for   PR 5725 at commit 70a39e4.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class ExecutorMemoryManager

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public final class TaskMemoryManager

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31185/
Test FAILed.

The most recent test failure looks like a possible double-free somewhere.  Here's what I saw when I ran that test locally:
[info] Test org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite.randomizedStressTest started
java(7099,0x111e21000) malloc: *** error for object 0x7fafec90e800: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

This error is non-deterministic.  Based on some local debugging with Thread.dumpStack(), I think that this was due to a race between BytesToBytesMap's finalizer and the tearDown() method in AbstractBytesToBytesMapSuite.  I think that we should remove the finalizers, since they add complexity and are no longer necessary since we have leak detection / cleanup at higher levels.

As part of UnsafeRowConverter refactoring, I realized that I forgot to add support for a whole bunch of primitives, including booleans.  These are trivial to add, so I'll go ahead and do that now.

Merged build triggered.

Merged build started.

Test build #31202 has started for   PR 5725 at commit eeee512.

LGTM. There are some issues, but we can easily address them in followup PRs.

Test build #31202 has finished for   PR 5725 at commit eeee512.

This patch fails MiMa tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class ExecutorMemoryManager

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public final class TaskMemoryManager

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31202/
Test FAILed.

Jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #31245 has started for   PR 5725 at commit eeee512.

Test build #732 has started for   PR 5725 at commit eeee512.

Test build #31245 has finished for   PR 5725 at commit eeee512.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public final class UnsafeFixedWidthAggregationMap

 public static class MapEntry

public final class UnsafeRow implements MutableRow

class UnsafeRowConverter(fieldTypes: Array[DataType])

public final class PlatformDependent

public class ByteArrayMethods

public final class LongArray

public final class BitSet

public final class BitSetMethods

public final class Murmur3_x86_32

public final class BytesToBytesMap

 public final class Location

 class Doubling implements HashMapGrowthStrategy

public class ExecutorMemoryManager

public class HeapMemoryAllocator implements MemoryAllocator

public class MemoryBlock extends MemoryLocation

public class MemoryLocation

public final class TaskMemoryManager

public class UnsafeMemoryAllocator implements MemoryAllocator

This patch adds the following new dependencies:

spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31245/
Test PASSed.

Ok merging in master. Let's make sure we address @marmbrus 's comments in followup prs.

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java, line 133 [r14] (raw file):
Please include writtenLength, unsafeRow.length and javaRow in the assertion so that we have more information when debugging.

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java, line 148 [r14] (raw file):
Condition doesn't match assertion message w.r.t. <=

unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java, line 23 [r14] (raw file):
Declare this class static ?

unsafe/src/main/java/org/apache/spark/unsafe/array/LongArray.java, line 26 [r14] (raw file):
nit: in-heap -> on-heap

unsafe/src/main/java/org/apache/spark/unsafe/bitset/BitSet.java, line 24 [r14] (raw file):
https://docs.oracle.com/javase/7/docs/api/java/util/BitSet.html doesn't serve your needs ?

unsafe/src/main/java/org/apache/spark/unsafe/bitset/BitSetMethods.java, line 75 [r14] (raw file):
Maybe use something similar to http://www.docjar.com/docs/api/sun/misc/Unsafe.html#getInt(Object, long) for speedup ?

Comments from the review on Reviewable.io

I'm not able to build after this went  in.  Any special requirements to build this?
[info] Compiling 15 Java sources to /home/tgraves/tgravescs_spark/unsafe/target/scala-2.10/classes...
[error] /home/tgraves/tgravescs_spark/unsafe/src/main/java/org/apache/spark/unsafe/PlatformDependent.java:22: warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future release
[error] import sun.misc.Unsafe;
[error]                ^
[error] /home/tgraves/tgravescs_spark/unsafe/src/main/java/org/apache/spark/unsafe/PlatformDependent.java:26: warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future release
[error]   public static final Unsafe UNSAFE;
[error]                       ^
[error] /home/tgraves/tgravescs_spark/unsafe/src/main/java/org/apache/spark/unsafe/PlatformDependent.java:43: warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future release
[error]     sun.misc.Unsafe unsafe;
[error]             ^
[error] /home/tgraves/tgravescs_spark/unsafe/src/main/java/org/apache/spark/unsafe/PlatformDependent.java:45: warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future release
[error]       Field unsafeField = Unsafe.class.getDeclaredField("theUnsafe");
[error]                           ^
[error] /home/tgraves/tgravescs_spark/unsafe/src/main/java/org/apache/spark/unsafe/PlatformDependent.java:47: warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future release
[error]       unsafe = (sun.misc.Unsafe) unsafeField.get(null);
[error]                         ^
[error] /home/tgraves/tgravescs_spark/unsafe/src/main/java/org/apache/spark/unsafe/memory/MemoryBlock.java:53: setMemory(long,long,byte) in sun.misc.Unsafe cannot be applied to (java.lang.Object,long,long,byte)
[error]     PlatformDependent.UNSAFE.setMemory(obj, offset, length, (byte) 0);
[error]                             ^
[error] 1 error
[error] 5 warnings
[error] Compile failed at Apr 30, 2015 2:52:04 PM [0.862s]

Did you have warning turned on into error?

not sure what your question is exactly, I think the last one is an error instead of warning.  I didn't specifying anything special in build command: mvn -Pyarn -Phadoop-2.4 -DskipTests clean package install

I'm looking into this issue now; these warnings can't be suppressed via the usual annotations mechanism; it requires a special compiler flag.  Planning to submit a patch soon; would be great to get your help in testing it.

Thanks, Also I should have specified, I'm running on a rhel6 box using jdlk6

Java 6's output:
$ javap sun.misc.Unsafe | grep setMemory
    public native void setMemory(long, long, byte);

So Spark is using an API that doesn't exist there.

On a side note, if Spark supports java 6, why are all jenkins builds running java 7?

@tgravescs I've opened #5814 to try to fix the compiler warnings (and error due to Unsafe.setMemory); it would be great if you could test out that commit and see whether it fixes things in your environment.

@vanzin There's an old JIRA for building w/ Java 6 on Jenkins; I've re-opened it: https://issues.apache.org/jira/browse/SPARK-1437


Support scipy.special with __array_ufunc__

Description
cupy.ndarray implements __array_ufunc__ interface, this allows NumPy ufuncs to accept CuPy arrays and it all just works. The majority of scipy.special functions are implemented as ufuncs and the same is true for cupyx.scipy.special. It would be great to support interoperability with SciPy's special functions module.
For example, a variant of the following code should work:
import cupy
from scipy.special import rel_entr, kl_div

p = cupy.array([1/2, 1/2])
q = cupy.array([1/10, 9/10])

print(rel_entr(p, q), sum(rel_entr(p, q)))
print(kl_div(p, q), sum(kl_div(p, q)))
assert type(rel_entr(p, q)) == cupy.ndarray
assert type(kl_div(p, q)) == cupy.ndarray

Running above code now would give TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'rel_entr'>, '__call__', array([0.5, 0.5]), array([0.1, 0.9])): 'ndarray', 'ndarray'.
Making the above code work correctly is just a two lines of code change:
diff --git a/cupy/core/core.pyx b/cupy/core/core.pyx
index 3d8b27daa..370a143d4 100644
--- a/cupy/core/core.pyx
+++ b/cupy/core/core.pyx
@@ -1317,6 +1317,7 @@ cdef class ndarray:
         numpy array.
         """
         import cupy  # top-level ufuncs
+        import cupyx.scipy.special  # special ufuncs
         if 'out' in kwargs:
             # need to unfold tuple argument in kwargs
             out = kwargs['out']
@@ -1331,7 +1332,7 @@ cdef class ndarray:
                 return NotImplemented
             name = ufunc.__name__
             try:
-                cp_ufunc = getattr(cupy, name)
+                cp_ufunc = getattr(cupy, name, None) or getattr(cupyx.scipy.special, name)
             except AttributeError:
                 return NotImplemented
             if name in [

Applying the patch and recompiling CuPy makes the example from the first post work:
In [1]: import cupy
   ...: from scipy.special import rel_entr, kl_div
   ...:
   ...: p = cupy.array([1/2, 1/2])
   ...: q = cupy.array([1/10, 9/10])
   ...:
   ...: print(rel_entr(p, q), sum(rel_entr(p, q)))
   ...: print(kl_div(p, q), sum(kl_div(p, q)))
   ...: assert type(rel_entr(p, q)) == cupy.ndarray
   ...: assert type(kl_div(p, q)) == cupy.ndarray
[ 0.80471896 -0.29389333] 0.5108256237659907
[0.40471896 0.10610667] 0.5108256237659906

In [2]: type(rel_entr(p, q))
Out[2]: cupy.core.core.ndarray

Is this something that would be acceptable to have in CuPy?
Additional Information
No response

@IvanYashchuk Just for my own curiosity: Do you know how scipy.special supports __array_ufunc__? Did they use the ufunc C APIs?

Yes, they use ufunc's C API (PyUFunc_FromFuncAndData) and they generate Cython code that binds Cython/C/C++/Fortran implementation to Python function.
See here https://github.com/scipy/scipy/blob/b6481610941b62d23bdd9a104840b58da35cac0b/scipy/special/_generate_pyx.py#L773-L777
and the docstring at the top of the file.


Print one page only

Hello, I'm glad to find your project vue-pdf-app. I've been looking for an out-of-the-box pdf-viewer, and now I'm integrating it into my open source project, zpan.
At present, a printing problem has been found. When you click to print, you can only print one page of content. I don't know where this problem is.
Configuration:

Web browser and it`s version: Chrome 92.0.4515.131
Operating system and it`s version: macOS 10.13.6
vue-pdf-app version: 2.1.0

Attachments (screenshots, links, etc)

only for print, download is normal.


Define strict mode in the nodejs api js

Edit: Moved to https://github.com/janeasystems/nodejs-mobile/issues/22 (according to "Reporting Issues" in the README)
Hi,
thanks for your awesome work on nodejs-mobile!
I noticed when adding the plugin to an android platform, building it and running it on a device, the console logs the following error messages:
Uncaught SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode plugins/nodejs-mobile-cordova/www/nodejs_apis.js:5
Uncaught Error: Module nodejs-mobile-cordova.nodejs does not exist. cordova.js:1470

To solve this it's necessary to put www/nodejs_apis.js into strict mode. With this change the plugin loads and logs the welcome message as expected.
NodeJs Engine Started
[cordova] received: Replying to this message: Hello from Cordova! index.js:32

Hope this helps.
Best regards

Hi @stoically
thank you for opening the PR.
We use one place to file all the issues, but it's fine to send PRs here, so I'm going to reopen it and we will merge it.
Thank you very much for your contribution.


Option to select language

[x] Portuguese
[ ] Spanish
[ ] English

Maybe will be more easy create dedicated versions for each version.


Mention recommended browser, Chrome, somewhere

We should mention that we recommend a particular browser, Google Chrome, version..., somewhere.
How about the following for now, before we take a closer look in 2020, along with a mobile and tablet friendly interface:
Welcome to the Source Academy playground!
The language Source is the official language of the textbook Structure and Interpretation of Computer Programs, JavaScript Adaptation. You have never heard of Source? No worries! It was invented just for the purpose of the book. Source is a sublanguage of ECMAScript 2016 (7th edition) and defined in the documents titled "Source §x", where x refers to the respective textbook chapter. For example, Source §3 is suitable for textbook chapter 3 and the preceeding chapters.
The playground comes with an editor and a REPL, on the left and right of the screen, respectively. You may customise the layout of the playground by clicking and dragging on the right border of the editor, or the top border of the REPL.
As of August 2019, we recommend the browser Google Chrome, Version 75 or higher, to visit the Source Academy.

I think chrome of firefox is fine also! but yes Chrome shall be our first class/supported browser!

I've added Firefox:
As of August 2019, we recommend the browsers Google Chrome, Version 75 or higher, or Firefox, Version 68 or higher, to visit the Source Academy.
Would that be better?

How about in the help area?

OK, so we could say:
Help
As of August 2019, we recommend the browsers Google Chrome, Version 75 or higher, or Firefox, Version 68 or higher, to visit the Source Academy. If you encounter issues with the Source Academy using these browsers, please use the following resources.

...
...
...


Add option to use default template

Instead of use #default_template create notes always with default template (choose it from plugin options).

Thanks for the idea!
I will investigate and see if this can be done. It should be relatively easy to implement.
I might implement it such that the first suggestion is the default template. That way you still have the possibility to choose another template if needed.

@dipsy  there is now a beta release  with this functionality. You can either install it manually or by using BRAT. It might take some time before I can make an official release with this functionality, but I will let you know when that happens.


Bump log4j from 2.19.0 to 2.20.0

Why are the changes needed?

log4j 2.20.0 release notes: https://logging.apache.org/log4j/2.x/release-notes/2.20.0.html

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

@bowenliang123 don't forget to update assignees and milestone after merging PR.

@bowenliang123 don't forget to update assignees and milestone after merging PR.

Trying to backport this change to 1.7.0. But failed to resolve the conflicts. Planned to update milestone after that.

Thanks, merged to master.


Sitting NPCs float

I know this is more likely an issue with Citizens, but whenever I make an NPC sit, they float above the block they're supposed to be sitting on.
/version Output:
This server is running Paper version git-Paper-85 (MC: 1.15.2) (Implementing API version 1.15.2-R0.1-SNAPSHOT)
Previous version: git-Paper-62 (MC: 1.15.1)
You are 2 version(s) behind

/version denizen Output:
Denizen version 1.1.2-SNAPSHOT (build 4572-DEV)
Author: The DenizenScript Team

/version citizens Output:
Citizens version 2.0.26-SNAPSHOT (build 1815)
Website: http://www.citizensnpcs.co
Author: fullwall

Screenshot: https://cdn.discordapp.com/attachments/315163488085475337/672945326927118349/2020-01-31_17.22.56.png

They don't float, they just disagree on where to sit. You might consider /npc sit -c or /npc sit --location x,y,z,world to better place the sitting position. Generally, sitting works best on a stair block or slab (stairs tend to look like chairs).


Something might still be wrong with K-Quant

When I run a 30B model (in this case upstage-llama-30b-instruct-2048.ggmlv3.q5_K_M.bin) the debug output in ollama talks about a 13B model size:

when running the same model with llama.cpp it outputs the correct size:

I tested with a 13B model and the output seems correct. Both models seem to work (as in generating output).

@nkoehring can you confirm the Modelfile you're using for upstage_llama_30b references the correct file on disk? You can also compare the SHA256 of the file with the blob path; they should match.

This is what I see using the same model (yes on Windows):
2023/07/26 20:12:50 images.go:208: [model] - C:\Users\michael_yang\Downloads\upstage-llama-30b-instruct-2048.ggmlv3.q5_K_M.bin
[GIN] 2023/07/26 - 20:25:05 | 200 |        12m14s |       <IP_ADDRESS> | POST     "/api/create"
llama.cpp: loading model from C:\Users\michael_yang\.ollama\models\blobs\sha256-00e2f30c83fb9230da3e54de681bfb9514b5b1f73a932fafcd7a01b058515e40
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 6656
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 52
llama_model_load_internal: n_layer    = 60
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 17 (mostly Q5_K - Medium)
llama_model_load_internal: n_ff       = 17920
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size =    0.14 MB
llama_model_load_internal: mem required  = 24238.87 MB (+ 3124.00 MB per state)
llama_new_context_with_model: kv self size  = 3120.00 MB

Notice the SHA256 hash is different: the one from your screenshot shows 6c6636176e21820adfc6781dee5b60d8385deea57fcc1d880930a401f339f95f (there may be some transcription errors) while my output shows 00e2f30c83fb9230da3e54de681bfb9514b5b1f73a932fafcd7a01b058515e40.

Closing this as can't reproduce. If there are more issues, please create a new issue


Add Slack support link in README's get started

@frascuchon  This is a very simple change to improve new users onboarding. As it's a readme change, no need to do much QA


Mobile Nav Bar

Do you think we can put the balance of the user next to the profile pic. It feels like a broken UI to have it under.

Hi Miki, this is how it's as of today on all dApps on Mobile. We'll out-scope it and fix it later for all dApps.

Closing it as we won't fix it.


Exact time for the world time target filter

Right now, when the target filter "time" for worlds is selected, and set to a specific time in ticks (eg. 18000 for midnight or 6000 for noon), no world is getting selected. I would love to have the possibility to chose an exact time to make an action happen exactly once a day.

The "time" filter currently only takes "day" and "night" as values. I'll add the ability to specify the time in ticks as well.

Done. A new version will be released shortly.


Working stream update : Implemented checks to identify document changes and introduced a transaction anchoring utility file.

Modified demo and added a file(updateDocument.ts) in the utils.
Delegate prepares a new content in the demo and calls the updateStream method in the updateDocument.ts. updateStream method in turn calls the updateFromContent method in the Document.ts.
This PR has been initiated in response to the reviews of - https://github.com/dhiway/cord.js/pull/108.

➤ YN0000: src/document/Document.ts(32,17): error TS6133: 'bool' is declared but its value is never read.
➤ YN0000: The command failed for workspaces that are depended upon by other workspaces; can't satisfy the dependency graph
➤ YN0000: Failed with errors in 1m 39s

➤ YN0000: src/document/Document.ts(32,17): error TS6133: 'bool' is declared but its value is never read.
➤ YN0000: The command failed for workspaces that are depended upon by other workspaces; can't satisfy the dependency graph
➤ YN0000: Failed with errors in 1m 39s

Resolved this


Implementing Binary Indexed Tree

Hi! I'm new to this repository and would like to try to implement the binary indexed tree!

@malfple thank you for nice implementation! It is merged in master now.


Tracing WG Meeting 2015-10-xx

Date/Time: TBD
Agenda and minutes can be collected in here and copied into this repo later:
https://docs.google.com/document/d/1KWFRzRrJJY8zx1rofZqLSh9KCWR77cV3OBZ7jTtffYw/edit?usp=sharing
Previous minutes: https://github.com/nodejs/tracing-wg/blob/master/wg-meetings/2015-09-25.md
Hangout on air for active participants: TBD
Hangout on air for viewers, event page: TBD
YouTube movie for observers and saved recording: TBD
Current invited participants, this list is not exclusive if you think you should also join:

TBD

(will need emails for google docs/hangout setup - send to<EMAIL_ADDRESS>Agenda discussion can happen in this thread.
(format copied from https://github.com/nodejs/LTS/issues/38)
(instructions for creating Hanouts On Air on the nodejs/node wiki)

So, funny thing happened when I brought up the question of having monthly calls, in the video at 5:10.  @ofrobots responds with "Monthly sounds good to me ...".  And then @natduca appears, and what I now identify as an echo of something I said 45 seconds early plays, instead of Nat talking.  Where I mention "I think we should be talking more frequently than that".  The context was from earlier, as in "more frequently than every seven months".
At the time, I thought it was Nat talking, asking for meetings more frequent than every month.
So weird.
Anyway, even though no one complained about meeting twice a month, I'm guessing once a month will be fine.
Thinking of targeting the end of October, which would put us at once a month.  @matthewloring had created a doodle for an Oct 5 meeting.  Matt, wanna create for more end of the month?
Discussion in this issue.  The agenda/minutes for for the next meeting is live, in this issue's description above.

Actually I am going to change my own opinion. Given that there is some momentum behind the discussion on trace-event, it might make sense to meet on Oct 5 after all while the knowledge about it is still fresh. Two weeks is long enough for people to digest the information and not too long to forget it.

Agh, no opinions from me. I'd look to @ofrobots to be point on this from goog/chrome side and he can loop me in as needed on specific calls.

I'd say generally monthly, but scale up to every two weeks as-needed.
It seems like there's more to discuss on the trace-event stuff, and we didn't really get anywhere with async_wrap, due to Trevor's absence. It might be good to run another hangout sooner rather than later.

Allrighty, let's try for week of Oct 5, with Matt Loring's doodle:

http://doodle.com/poll/zwk6c8exh8cn92rn

Contact me or Matt if you need the admin bit of the doodle.
Folks, start you're doodling!
@AndreasMadsen @Qard @Wraithan @bnoordhuis @brycebaril @chiniforooshan
@dberesford @domenic @fmeawad @groundwater @hayes @lucamaraschi @lykkin
@matthewloring @mikeal @ofrobots @othiym23 @paulirish @piscisaureus @pmuellr
@rmg @rvagg @sam-github @thekemkid @thlorenz @trevnorris @yunong @yury-s

You can also just mention @nodejs/tracing to catch everyone in the team. :)

@pmuellr Would you mind adding me to the README?

@yunong I would, but really you should be added to the Tracing team as well.  And I can't do that, or don't know how to.
Suggest you do an edit right from the GH page for the README, create a pull request from that, and add a comment that you want to be added to the Tracing team.

You can request to be added to the team here: https://github.com/orgs/nodejs/teams/tracing
You'll have to ping someone in there with the owner bit to approve you.

Looking at the doodle, we have 9 folks who have responded, and there are 2 times that have 7 check marks (the most check marks).  Those are:

Wed 7 at 1:00 PM eastern - world time
Thur 8 at 4:00 PM eastern - world time

@rvagg and @Qard can't make the Wednesday time
@rvagg and @lucamaraschi can't make the Thursday time
@rvagg / @Qard / @lucamaraschi, thoughts?
Given no other info, guess I'd opt for Wed, it's first.

Also need to work on the agenda.
Presumably @trevnorris could give us an update on AsyncWrap, and @ofrobots on v8-trace.
Would be nice to have the docs section of the wiki fleshed out.  If you know something, write something (or whatever :-) )

I filled in the doodle. Sorry for the late response.
@pmuellr you should add the doodle link to the overview, I missed it in the comments.

please don't let this hinge on me in any way, I'll make it if it lines up with my availability (I probably shouldn't have filled out the doodle!)

Wednesday at 1PM EST is not great for me, but I can make it work, if necessary. I'll have to cut out by 2PM though.

OK, looks like Wed 1PM is the date - much thanks to @Qard for squeezing it in.  I'll set up the HoA, update dates/links in the issue description, etc.
Still need agenda bits - see comment from 3 days ago

Looks like everyone on the Doodle (incl Rod, for completeness) has been sent an invite to the hangout.
Except @fmeawad, since I don't have, or misplaced, his email address.

Its<EMAIL_ADDRESS>

@pmuellr I did not get the invite :(

@pmuellr me neither

Hmmm, maybe I forgot to click something to actually send an invite.  Please check the event page and make sure you're listed on the invite list; I see: Patrick Mueller, Fadi Meawad, Yunong Xiao, Matt Loring, Ali Sheikh, Andreas Madsen, Stephen Belanger, Luca Maraschi, Rod Vagg, Trevor Norris.

I've given up on trying to use the Hangouts invite system, for now, and have curated a list of emails for folks in the tracing team, and other folks who have expressed an interest in the tracing wg.  If you are reading this, and didn't get an email from my gmail account titled "Node.js Tracing WG Meeting 2015-10-07", send me your email offline, to<EMAIL_ADDRESS>

Added overview of AsyncWrap issues as requested: https://github.com/nodejs/tracing-wg/issues/29
Added AsyncWrap HTTP issue as required: https://github.com/nodejs/node/issues/3241

Thanks all for participating in the meeting today! I feel like we made some great progress on finally establishing a direction for the tracing work.
I'm going to close this and we can continue discussion in the other new issues. :smile:

I've added minutes for this meeting here - https://github.com/nodejs/tracing-wg/blob/master/wg-meetings/2015-10-07.md


adding Junkabal.edu.gt

adding Junkabal.edu.gt

@luciavelasquezcjk First stage of verification passed, official domain is junkabal.edu.gt as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@luciavelasquezcjk Pull request merged. Thank you.


GH-32276: [C++][FlightRPC] Align buffers from Flight

Rationale for this change
Ensure buffers from Flight are at least 8-byte aligned to meet general expectations of downstream users.
What changes are included in this PR?
Manually align buffers after deserialization.
Are these changes tested?
Yes
Are there any user-facing changes?
Flight data will be aligned, at the cost of an additional copy.

Closes: #32276

I inlined some stuff from @westonpace's PR but I'd rather wait and be able to use those utilities directly. That said it demonstrates the problem and the solution.
Given more time we might be able to avoid this, but I'd need to dig deeper into Protobuf & do some benchmarking. (In particular: instead of going gRPC slices -> CodedInputStream -> Arrow buffers, I'd rather directly go from gRPC slices -> Arrow.)

Given the nature of the gRPC API there's not a great way to customize it at runtime. If someone complains about this in the future we can point them here then. Possibly we can make it a compile-time toggle, though that's effectively useless to most users.
@westonpace @rtpsw were the most recent to mention this. If there's not a real desire for this then I'll close the PR.

This could be settable using an env var. But I don't think we should realign buffers by default. This can decrease performance and increase memory fragmentation for unsuspecting users.

Now that we have a workaround enabled in Acero I don't have any urgent need for this.  I'm fine with waiting for it to become a problem (and it may never do so).

@pitrou

But I don't think we should realign buffers by default.

The arrow standard requires it.

Implementations are recommended to allocate memory on aligned addresses (multiple of 8- or 64-bytes) and pad (overallocate) to a length that is a multiple of 8 or 64 bytes. When serializing Arrow data for interprocess communication, these alignment and padding requirements are enforced.


Diff color is too dim - see picture inside

In the picture above there is just one byte difference but it is not very visible (3e vs 40)


Point out optional dependency for output coercion

Using (client/get ... {:as :json}) without the [chesire] dependency, will keep the :body as a String rather than parse the body as json. Much later in the README there is a section on Option Dependencies that refers to this. This note is to save other readers some time.

Merged, thanks!


node_modules/wdio-cucumberjs-json-reporter/dist/cjs/index.d.ts' is not a module

Environment (please complete the following information):

**Node.js version: 18.16.0
**NPM version: 9.5.1
**webdriver.io version: 8.10.0
**@wdio/cucumber-framework version: 8.10.0
**wdio-cucumberjs-json-reporter version: 5.1.0

Config of webdriver.io and the reporter
export const config = {     runner: "local",     specs: ["./../src/features/**/*.feature"],     exclude: [],     maxInstances: 1,     capabilities: [],     logLevel: "warn",     bail: 0,     baseUrl: baseUrl,     waitforTimeout: 20000,     connectionRetryTimeout: 60000,     connectionRetryCount: 3,     services: [],     framework: "cucumber",     reporters: [         [             "cucumberjs-json",             {                 jsonFolder: reports.jsonReportFolder,             },         ],     ],     cucumberOpts: {         requireModule: ["tsconfig-paths/register.js"],         require: ["./src/steps/**/*.ts"],         backtrace: true,         dryRun: false,         failFast: false,         format: ["pretty"],         colors: true,         snippets: false,         source: false,         profile: [],         strict: true,         tagExpression: "",         timeout: 60000,         ignoreUndefinedDefinitions: false,     },     autoCompileOpts: {autoCompile: true,         tsNodeOpts: {             transpileOnly: false,             project: "tsconfig.json",             files: true,         },     },     ...hooks, };
Describe the bug
I updated my automation project from wdio v7 to wdio v8. Everything works fine except the wdio-cucumberjs-json-reporter package.
After the update, I got this error:

This is my tsconfig file:
{     "compilerOptions": {         "target": "ES2022",         "module": "CommonJS",         "moduleResolution": "node",         "noImplicitAny": true,         "strict": true,         "noEmit": true,         "removeComments": true,         "esModuleInterop": true,         "skipLibCheck": true,         "forceConsistentCasingInFileNames": true,         "pretty": true,         "allowSyntheticDefaultImports": true,         "resolveJsonModule": true,         "types": [             "node",             "expect-webdriverio",             "@wdio/globals/types",             "@wdio/cucumber-framework"         ],         "baseUrl": ".",         "paths": {             "@base/*": [                 "src/support/pages/@base/*"             ],             "pages/*": [                 "src/support/pages/*"             ]         }     },     "include": [         "./src/**/*.ts"     ],     "compileOnSave": false,     "typeAcquisition": {         "enable": false     } }

Just pushed a fix and released it. Thanks for reporting.

Hi @christian-bromann
Thanks.
However, I think it is not completely fixed, because the methods are not found:

@joao-motork can you provide a minimal reproducible example?

I am attaching the full project here:
boilerplate.zip
you can find the call to the "attach" method in the hooks.ts (src/support/hooks.ts)

Hi @christian-bromann
When I created the zip file of the project I forgot to revert that last minute change (some random trials that I was doing).
Anyway, I've just updated wdio-cucumberjs-json-reporter to version 5.1.2 and it seems to work now.
Thanks


Add Swedish podcasts

it's still the wrong name: casts/free-podcasts-screencasts-se.md  will submit a change


Added from/to height query filters to the transaction search endpoints

Closes https://github.com/nemtech/catapult-rest/issues/469

note that this aims to allow the clients to be able to filter transactions by height (period) range which is what the architecture offers
as you know, the chain does not contain explicit real world timestamp data, thus if user experience needs to be improved, it is in the hands of the SDK/clients to offer a way to convert height to timestamp (probably involving timestamp fields and nemesis block creation data)

this is ready but waiting for fernando to give green light (he asked us to wait to merge this because his schemas would be regenerated or something)


Do the targetencoding and leave one out encoding work with both classification and regression problems?

Do the targetencoding and leave one out encoding work with both classification and regression problems?

They will run, but you may not get great results, they are intended for classification problems I believe.


🐛 [BUG]: Hiding/Unhiding Nodes/Edges can be expensive and block the main thread

Is there an existing issue for this?

[X] I have searched the existing issues and this is a new bug.

Current Behavior
So I'm using Vue Flow to render components as nodes for a data graph. They are fairly complex in comparison to the examples shown on the docs. I've made it so I can hide all descendant nodes by clicking a button on a parent node. The action of hiding descendants is fast but unhiding them is very slow and locks up the main thread.

When unhiding elements from the root node, the operation took over 100ms to complete. The flame graph on the perf monitor is pretty deep here.
Expected Behavior
Unhiding should be fairly quick. Would it be possible to do some kind of memoization of previously hidden elements instead of completely un-rendering them? That, or hide them using CSS?
The hide operation is handled by Vue Flow itself so I don't really have control over it too much.
Steps To Reproduce
No response
Relevant log output
No response
Anything else?
Also, I'm not able to use Vue Flow's built in node parent referencing capabilities because it affects the way layouts are computed in an undesirable way.

The issue seems to be performance which could be anything really, since you haven't presented any repro or code-snippets I don't know what your application looks like and if you're not doing somehow causing some excessive re-renders unknowingly.
This sort of report really needs a reproduction.
Looking at the screenshot, re-mounting 5 nodes should not have any noticeable impact on performance whatsoever, even if they have a bit more complexity, which they don't look like they do in your img but who knows - could it be you're doing some heavy computations in those nodes? Is that necessary?
Anyway, I'm moving this to discussions since there's no actual bug described here; possibly a feature/refactor request for how visibility of nodes could be handled instead :)


[wip] push docker images to docker hub

Signed-off-by: Wataru Ishida<EMAIL_ADDRESS>

OK to test


Visual enhancement of the Egress Rules section

Is your feature request related to a problem? Please describe.
In the rules section, when we use a lot of rules it becomes complicated to visualize them properly.
Describe the solution you'd like
It would be nice to be able to move them around when they've already been created, and to be able to group them together.
Creating separators and naming them would also be good for visual clarity, as in many firewall software interfaces.

Thanks for the request. We'll be announcing 1.0 soon with major changes in this area. The main one being that rules will become Policies and much more simplified, tied to user Groups, and easier to manage.

We're aiming for 1.0 beta release in a month or so, with general availability due in Q4


Network trained with 4 GPUs and 54 epochs has worse performance

I trained Neucon on 4 RTX5000 (16G) for 54 epochs. However, the evaluations on the Scannetdataset test split are worse than your released model trained for only 47 epochs.
Here are the results from my trained model (54 epochs) :
AbsRel 0.072
AbsDiff 0.117
SqRel 0.037
RMSE 0.207
LogRMSE 0.120
r1 0.927
r2 0.960
r3 0.973
complete 0.861
dist1 0.066
dist2 0.179
prec 0.568
recal 0.349
fscore 0.429
Here are the results from your released model (47 epochs):
AbsRel 0.065
AbsDiff 0.098
SqRel 0.038
RMSE 0.197
LogRMSE 0.113
r1 0.933
r2 0.961
r3 0.974
complete 0.892
dist1 0.053
dist2 0.133
prec 0.687
recal 0.473
fscore 0.558
Do you have any clues for this? Thanks a lot!

May I ask if you train the neucon on the whole ScanNet data set (with over 1Tb in size) or the preprocessed version (about 5.6Gb)?

May I ask if you train the neucon on the whole ScanNet data set (with over 1Tb in size) or the preprocessed version (about 5.6Gb)?

Please specify which "preprocessed version" you are referring to.

May I ask if you train the neucon on the whole ScanNet data set (with over 1Tb in size) or the preprocessed version (about 5.6Gb)?

Please specify which "preprocessed version" you are referring to.

Hi Jiaming Suen,
Thanks for reply and your impressive work on 3D reconstruction. May I ask, what's the size of scannet you used for training NeuralRecon. The entire scannet_v2 (1.2TB) or the 5.6GB "scannet_frames_25k.zip"?
p.s.  In the python script provided by Prof. Dr. Angela Dai, "PREPROCESSED_FRAMES_FILE" refers to the 5,6GB training set. Sorry,I didn't make it clear.
Best regards,
Po

It should be scannet_v2 (1.2TB).

I have the similar problem. I trained the network on 8 GTX 2080 Ti with batch size of 1 on each GPU. Then I tested the results of models trained after 25, 30, 35, 40, 45, 50, 60, 70 epochs. The best model only gets a performance of abe_rel=0.068 and fscore=0.482, which is far from the results in your paper.
Also, the results from your released model (47 epochs) is:
AbsRel 0.065
AbsDiff 0.099
SqRel 0.038
RMSE 0.197
LogRMSE 0.113
r1 0.932
r2 0.961
r3 0.975
complete 0.892
dist1 0.053
dist2 0.135
prec 0.687
recal 0.471
fscore 0.557
which is the same as that reported by @ZuoJiaxing while different from that in the paper.
May I ask the training setup of yours? e.g. the batch size, the numbers of GPUs, the learning rate
Many thanks!


higher tiles resolution

Hi! Once again, thanks for the great work. Is there a way to use better resolution of tiles, i.e. downloading more tiles and showing them scaled down, more packed. The OpenStreetMaps and even MapBox tiles look too pixelized.

Okay, apparently it works great, if I get retina tiles from MapBox.


Disable default provider for Openssl-3

I am using Openssl-3 with the FIPS provider in my openssl.cnf file. I understand that the default provider is built into libcrypto and is not provided like a shared object. While testing, I realized that I can always override the openssl.cnf file and load and activate the default provider which allows me to use non-FIPS algorithms like ChaCha20. I can use the following APIs to do this.
OSSL_PROVIDER_load(NULL, "default");
EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
EVP_CIPHER *temp = EVP_CIPHER_fetch(NULL,"chacha20","provider=default");

Is it possible to make openssl.cnf file the source of absolute truth for which provider is being used for cryptographic implementation. So if my openssl.cnf file looks like this:

config_diagnostics = 1
openssl_conf = openssl_init
.include fipsmodule.cnf
[openssl_init]
providers = provider_sect
alg_section = evp_properties
[provider_sect]
fips = fips_sect
base = base_sect
[base_sect]
activate = 1
[evp_properties]
default_properties = fips=yes

I can be sure that there is no way for me to load and activate any other provider. Is there an API or a config which will stop me from loading another provider at runtime?

There is no API call to do what you want.  A new provider can always be loaded.
If you want to force the FIPS provider's algorithms you can set the default property query: EVP_set_default_properties(NULL, "fips=yes");

Closing since this is marked as resolved.


Fixed documentation of SimpleRope

Description of change
The documentation of SimpleRope was referring to the older Rope.
Pre-Merge Checklist

[x] Documentation is changed or added

Codecov Report

Merging #6010 into dev will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              dev    #6010      +/-   ##
==========================================
+ Coverage   70.54%   70.55%   +<.01%
==========================================
  Files         208      208
  Lines       10530    10530
==========================================
+ Hits         7428     7429       +1
+ Misses       3102     3101       -1

Impacted Files
Coverage Δ

packages/mesh-extras/src/SimpleRope.js
27.27% <ø> (ø)
:arrow_up:

packages/ticker/src/Ticker.js
87.5% <0%> (+0.78%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b47c4bb...73afeaa. Read the comment docs.


Convert Traces Connected to Net-Labeled Ports to Net Labels

All traces connected to ports with a netlabel should also be explicitly labeled with the same net name.
Example:

CC: @seveibar
what do you think about this?


fix(@angular/cli): correct blog link to blog.angular.io

Dupe of #7387?

Closing as a Duplicate of #7387


Search component incorrect right offset of close icon with "lg" mod

https://github.com/carbon-design-system/carbon-components/blob/8c124924d5549850703fda87dde079b91964dd61/src/components/search/_search.scss#L88
rem(100px) looks good with icons:

but without:

@carbon-design-system/design should the large search be used without icons? As you can see above, not supported in code.

Large search without icons should be a supported pattern.


change background color by code

I used setbackgroundcolor but not working

I just used forked library in my project and implement needed methods(( but it's not right


Call :erl_tar.extract/2 with charlist cwd option

This function call is working when cwd is a binary right now,
but it is documented that cwd should be a charlist (string() in erlang type names).
Alternatively I can send a PR to OTP to update the spec to allow a binary there.
This PR is similar to https://github.com/elixir-lang/elixir/pull/9989

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart:


Effectively use fzf

Especially for completions...

fzf new release 0.48.0 looks like much useful (Maybe, walk can be completed in fzf...)
Shell Script integrations are included, except PowerShell(But if it include like https://github.com/kachick/PSFzfHistory, I'll be happy)
https://github.com/junegunn/fzf/releases/tag/0.48.0
The ENV style behavior customizing may fit for direnv 🤔 (Or not fit? The config will be shared each developpers)
Updating PR in nixpkgs are https://github.com/NixOS/nixpkgs/pull/295978

Especially for completions...

forgotten 😋


Per-Instance netboot images + a slew of bugfixes

Rename all the instances to match their new names
Add support for c2.medium.x86, x1.small.x86, m2.xlarge.x86
metadata generator now works for 18.03
multipath modules are loaded by default
ipv6 works out of the box now
disable dhcpc since ... it doesn't work ... after the first install
private networking fixed with a <IP_ADDRESS>/8 route
default packet-specific  nameservers setup
dumpkeys logs errors if something is broken
GRUB's interface is now usable on all but the ARM machines, bringing those famous boot-based  emergency rollbacks

note: s1.large is broken because the root device moves around too often
note: the arm machine is still sometimes broken :(
massive thank-you to @mmlb for his help on a bunch of this.

:+1:


Usage of second frame and it's depth in Object_motion_net

Hello! In this line, shouldn't 'motion_features' contain the rgb and predicted depth of second frame as well? So index 0 would be 1 in the mentioned line, correct? Please feel free to correct me in case I've understood it incorrectly

Thanks for raising the issue. I will modify the index and update it from 0 to 1 inorder to use the second frame as well.

Thanks for the quick response. Another question wrt training; as you are already working with kitti, could you tell me how many epochs (approximately) have you been training before you get good results?

https://github.com/chamorajg/pytorch_depth_and_motion_planning/blob/master/train.py#L72. I have updated the code here. I am working on KITTI during my free time. Generally, it takes around 100 epochs to get good results.

I am closing this issue as the bug was fixed. If you find any other bug feel free to open one.


Provide simple Kubernetes YAML

For Kubernetes deployments, it may be nice to provide a simple YAML. Not hard to make one manually, but harder than copy+paste :slightly_smiling_face:
Below shows a good start.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lgtm
spec:
  selector:
    matchLabels:
      app: lgtm
  template:
    metadata:
      labels:
        app: lgtm
    spec:
      containers:
      - name: lgtm
        image: grafana/otel-lgtm
---
apiVersion: v1
kind: Service
metadata:
  name: lgtm
spec:
  selector:
    app: lgtm
  ports:
  - name: ui
    port: 3000
  - name: grpc
    port: 4317
  - name: https
    port: 4318

Good idea - a PR would be highly welcome.
Please mention that this setup is not intended for production use.

:+1: yes, if you have the time please open an PR. For reference, here's one I made for a demo, looks pretty similar to your example :) https://github.com/fstab/2024-03-meetup-beyla-demo/blob/main/monitoring-backend.yaml


Add an implementation plan template

Description

This PR adds an implementation plan template, similar to the project proposal template and based on the implementation plan description of our project notes document.
I, personally, find it difficult to start with an empty page, so hopefully this will be a useful guide for folks writing implementation plans! I'm sure this could be structured better or that I might be missing some aspect; any feedback is welcome.
Checklist

[x] My pull request has a descriptive title (not a vague title like
Update index.md).
[x] My pull request targets the default branch of the repository (main) or
a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[ ] I added or updated tests for the changes I made (if applicable).
[x] I added or updated documentation (if applicable).
[ ] I tried running the project locally and verified that there are no visible
errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

I'll also try this with my recognition proposal that I intend to publish tonight!

If the overview is a brief one or two sentences description, we might want to also add a section for a longer description of the implementation plan. One more section that could be useful is the "Expected outcomes"

I can change the description of "overview", and add an "expected outcomes" section.


'LltJbd' object has no attribute 'min_battery_voltage'

Hello Team!
I have only seen this with 1 other user, and it was reported after a wifi dongle had been removed, but a resolution was never posted.
I have wifi built in, however I have been trying several different USB to serial adapters in the process of trying to get things to work. My latest is a dual serial startech with a prolific chipset.
I also own two Cerbo Gx's but one I damaged while trying to hook up tank sensors, so I am sensitive to that, and wanted to get this all working in my lab before I take it out to my motorhome at my farm 2 hours drive away.
The Venus unit is a RPI4 running v2.82-large-30
The unit also has a touch screen and is using HDMI 1
The BMS is a JBD in a 4s config with 4 x 100AH LIFEPO4 EVE Cells
Log when running:
cat /data/log/dbus-serialbattery.ttyUSB0/current | tai64nlocal
_2022-06-23 08:24:48.445116500 INFO:SerialBattery:Starting dbus-serialbattery
2022-06-23 08:24:48.445610500 INFO:SerialBattery:dbus-serialbattery v0.12
2022-06-23 08:24:48.446815500 INFO:SerialBattery:Testing LltJbd
2022-06-23 08:24:48.844155500 INFO:SerialBattery:Connection established to LltJbd
2022-06-23 08:24:48.867069500 INFO:SerialBattery:DeviceInstance = 3
2022-06-23 08:24:48.867531500 INFO:SerialBattery:com.victronenergy.battery.ttyUSB0
2022-06-23 08:24:49.256784500 Traceback (most recent call last):
2022-06-23 08:24:49.256789500   File "/opt/victronenergy/dbus-serialbattery/dbus-serialbattery.py", line 115, in
2022-06-23 08:24:49.256793500     main()
2022-06-23 08:24:49.256795500   File "/opt/victronenergy/dbus-serialbattery/dbus-serialbattery.py", line 100, in main
2022-06-23 08:24:49.256799500     if not helper.setup_vedbus():
2022-06-23 08:24:49.256801500   File "/opt/victronenergy/dbus-serialbattery/dbushelper.py", line 76, in setup_vedbus
2022-06-23 08:24:49.256805500     self.dbusservice.add_path('/Info/BatteryLowVoltage', self.battery.min_battery_voltage, writeable=True)
2022-06-23 08:24:49.256873500 AttributeError: 'LltJbd' object has no attribute 'min_battery_voltage'
2022-06-23 08:25:10.477830500 INFO:SerialBattery:Starting dbus-serialbattery
2022-06-23 08:25:10.478318500 INFO:SerialBattery:dbus-serialbattery v0.12
2022-06-23 08:25:10.479511500 INFO:SerialBattery:Testing LltJbd
2022-06-23 08:25:10.874135500 INFO:SerialBattery:Connection established to LltJbd
2022-06-23 08:25:10.895664500 INFO:SerialBattery:DeviceInstance = 3
2022-06-23 08:25:10.895975500 INFO:SerialBattery:com.victronenergy.battery.ttyUSB0
root@raspberrypi4:~#
Settings inside /data/etc/dbus-serialbattery/utils.py:
_# Constants - Need to dynamically get them in future
DRIVER_VERSION = 0.12
DRIVER_SUBVERSION = ''
zero_char = chr(48)
degree_sign = u'\N{DEGREE SIGN}'
Cell min/max voltages - used with the cell count to get the min/max battery voltage
MIN_CELL_VOLTAGE = 2.6
MAX_CELL_VOLTAGE = 3.55
battery Current limits
MAX_BATTERY_CURRENT = 100.0
MAX_BATTERY_DISCHARGE_CURRENT = 100.0
Charge current control management enable (True/False).
CCCM_ENABLE = True
Daly settings
Battery capacity (amps) if the BMS does not support reading it
BATTERY_CAPACITY = 100
Invert Battery Current. Default non-inverted. Set to -1 to invert
INVERT_CURRENT_MEASUREMENT = 1_
My understanding is that this occurs when it can not accurately get a cell count due to voltage.
Could it be that my BMS is in a cell overvolt state and thus communicates differently? especially if in a mosfet disconnect state?
I did run a slow and low amp (7 watts) charge on it before hand to help allow everything to balance out.
As I don't see this reported with many other users, I am sure it is something on my end, like GPS, or the touch screen fighting over serial ports.
I hope my logs and info help narrow things down.
Thanks!

Daly BMS here - seeing the same issue on a raspberry PI 4

@IamSierraCharlie which version of the driver are you using when you get this?
Try the latest beta v0.12b3 and also post the battery settings from the log file when it starts

@Louisvdw thanks for your message.
I'll try as you suggest; Although I think my issue is either the USB to Serial converter or the fact that I am trying to connect to the Daly BMS via RS485.  I have installed the USB to serial driver in Windows and run the Daly software - still no dice.  I have a USB to UART adaptor on the way.  I also installed Victron S/W on a raspberry Pi 2 today - same fault but I dont think its software.  If I find some compelling info that suggests otherwise, I'll report back.  Thanks again for responding!

Just to close out my comment above - new USB to RS485 adapter turned up today.  Swapping this with the one I have fixed the issue I was having.  Id suggest the old one "kind of worked", but the new one actually works properly.


Fixed uninitialized xfade in AnimationNodeTransition

This change fixes xfade being uninitialized in AnimationNodeTransition.
It was causing transition nodes in animation tree to be sometimes loaded with a small negative value for xfade. In this case, an input set with auto-advance couldn't switch to the next input when completed.

Cherry-picked for 3.1.1.


feature/dependabot-config

feature/dependabot-config

At the moment dependabot does not support kotlin gradle file. See https://github.com/dependabot/dependabot-core/issues/2238.


Dashboard status display not implemented yet

I love how we use the 🌔 emoji 👍

I rephrased the title, as currently we just have a static emoji there 👍

🎉


Issues building on a machine with PowerShell Version 3

When trying to build on a system with PowerShell Version 3 there are several build errors.  There are several methods that are referencing types in the new V5 System.Management.Automation DLL.   Adding several additional conditional compilation directives allows this to be successfully built on a machine with PowerShell Version 3.

Solution configurations should be updated so all projects build with the appropriate PSV3 build configuration.  The ScriptAnalyzerBuiltinRules project is not set to the appropriate PSV3 configuration.
Several methods reference types only available in the updated System.Management.Automation DLL.

Engine/VariableAnalysisBase.cs

Add conditional compilation directive (#if !PSV3) around internal static TypeDefinitionAst FindClass(Ast ast, string name) method
Add conditional compilation directive (#if !PSV3) around internal static TypeDefinitionAst FindClassAncestor(Ast ast) method

Engine/Helper.cs

The Configuration value does not exist in the PSV3 version of the CommandTypes enum.  Adjust method signature with conditional compilation directive to remove the unsupported enumeration value. public CommandInfo GetCommandInfo(string name, CommandTypes commandType = CommandTypes.Alias | CommandTypes.Cmdlet | CommandTypes.Configuration | CommandTypes.ExternalScript | CommandTypes.Filter | CommandTypes.Function | CommandTypes.Script | CommandTypes.Workflow)
Add conditional compilation directive (#if !PSV3) around internal List<RuleSuppression> GetSuppressionsClass(TypeDefinitionAst typeAst) method

Is there any update on this? Our build machines are hitting this same issue as we do not run PS5 on them. @mattgartman Did you find a fix/workaround?

Raghu,
If you follow the steps I outlined it will build fine on a machine with PoSHv3 (assuming PoSHv4) as well.  I didn't submit a pull request as I wasn't sure if that was appropriate or not.

@mattgartman I can confirm that the changes on your fork work on PowerShell Version 4


Pull in transfers as location for encounters

The mimc4fhir team pulled in mimic_core.transfers. They set the transfer information in the Encounter.
What we can do then:

Create careunit Locations, these will be referenced in the Encounters
Pull in Location history to Encounters based on time window of Encounter

I created MIMIC_Encounter_Transfers but the only useful information we were storing in the encounter was the timing and careunit name. Both of which could be readily stored in the base Encounter.location
We are reverting back to the idea of careunits as Location resources. This allows the careunit transfer history to be stored for both MIMIC_Encounter and MIMIC_Encounter_ICU. This will include the time periods in each careunit.


scatterplot Viz banner, behaviors, etc.

This will address scatterplot Viz works: https://github.com/VEuPathDB/web-eda/issues/1423 and https://github.com/VEuPathDB/web-eda/issues/1424.
Screenshots are:

banner for no smoothed mean

when pressing Why? at banner

log scale and smoothed mean/best fit (no plot is drawn as requested)

when pressing Why? at banner

@bobular Thank you for your reviews and tests 👍

Also @danicahelb - are we going to make Lineplot consistent? It currently uses snackbars to inform the user that logscale wasn't appropriate (and gets switched off) when error bars were switched on.

@bobular yes Lineplot Viz is currently using snackbars for log scale stuffs as a part of the previous ticket, https://github.com/VEuPathDB/web-eda/issues/1291 where both Scatterplot and Lineplot Vizs were dealt with. As you said, it may be better to have a consistent UI for this too.

@bobular we discussed your issue about needing to change your mouse position in the 12/2/22 eda ux meeting. I made these 2 tickets to address this:
https://github.com/VEuPathDB/EdaNewIssues/issues/497
https://github.com/VEuPathDB/EdaNewIssues/issues/496

@bobular @moontrip we also discussed the inconsistency in line plots with the snackbar. i made this ticket to address that:
https://github.com/VEuPathDB/EdaNewIssues/issues/498


Github Authorization not working

ATM, on https://launch.prod-preview.openshift.io/launch/, github authorization is not working. When clicking Log In & Authorize Account, it does not allow me to go forward.

@edewit @ia3andy any idea?

This is also broken in minishift.
To reproduce:

Install using: https://launcher-preview.fabric8.io/docs/minishift-installation.html#installing-launcher-tool_minishift
Create a booster using the GH flow
Notice it prompts you to authorize when it should already be connected to GH.
If you attempt to authorize, it redirects you to the start page

If you use the v20 version of the template, everything is fine.

@edewit this is the commit that broke it: 135828011a25f97799e734003729d4b4c2bcf21c. I am reverting it now

thank you @gastaldi, LGTM


Highlight the next prayer in all masjids

This basically answers the main question: Where And When to pray! We can even put this info above all tables.

Related to #5, if the grouping is by masjid we highlight the next prayer in all tables. If the grouping is by prayer, we show the next prayer as the first group, followed by others in sequence. For example, if the next prayer is Asr, the order will be:
Asr - Maghrib - Isha - Fajr - Zuhr.

@hammady We will have the compact layout as a separate component, possible to handle each in a different way.

This will be closed after adding highlights for jumaas, I will do #63 first)


Add TravisCI release automation

FWIW this will require some tags to be pushed through for testing purposes.

LGTM

Codecov Report

Merging #1 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master       #1   +/-   ##
=======================================
  Coverage   31.32%   31.32%
=======================================
  Files           2        2
  Lines          83       83
=======================================
  Hits           26       26
  Misses         56       56
  Partials        1        1

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update efda394...5c0c2fe. Read the comment docs.


What do you think about adding the spyTag?

spyTag :: forall a. DebugWarning => String -> a -> a
spyTag tag a = traceAny (unsafeSet tag a {}) \_ -> a

The normal spy already works this way now.

ah nice!


Add GOPASS_NO_SYMBOLS to strip output from any emojis

Fixes #1902
RELEASE_NOTES=[ENHANCEMENT] Add GOPASS_NO_SYMBOLS
Signed-off-by: Dominik Schulz<EMAIL_ADDRESS>

Oh my. Yes, this might get us into trouble. In a world where UTF-8 is a thing this might really happen ...

Yup, I'd be truly surprised if there aren't people out there using smileys in their passwords whenever they can nowadays 😅


Implement status pings

A status ping is done by POSTing a key to /status.json.  It returns a JSON object, containing a boolean called should_standby which indicates whether the instance which pinged should be in standby mode.

The preferred instance functionality should be manual, ideally.

@ArtOfCode- Yeah, I agree.  I'll implement this when I have time (probably this afternoon or evening, PDT); see my messages staring here


User based query permission

As detailed in this question, https://stackoverflow.com/questions/45320643/django-sql-explorer-user-based-query-access can we authorize different queries based on User or User Groups?
For example, for an organisation, there are 2 groups - workers and managers.
Workers group should only get access to query1 and query2. Managers group has access to all queries.

How is this related to django-admin-tools ?!


Timeline Section

A timeline section for a hackathons website to display the schedule of the events.

Can I work on this @Sanika2103 ?


Unexpected behavior in recursive data structure declarations

See: http://share-elm.com/sprout/55afc18ae4b06aacf0e8b605.
It seems like cyclic declarations are not always doing expected things. The output mechanism initializes b first, so it resolves a as undefined and this leads to crashes. Swap the order of declarations or which node you are inspecting and it works.
I'm unsure on how tractable this issue will turn out to be given the current output mechanisms, but it'd be very nice to have supported. But even without that support, there should be a consistent cycle detector and a compile error for it.
Note this related issue in an elm library: https://github.com/Dandandan/parser/issues/15#issuecomment-123765681

This is Elm being a non-lazy language. Such recursive value (rather than function) definitions are idiomatic Haskell, but not idiomatic Elm.
Enabling such recursive definitions in general would require a lot of overhead compared to the current implementation: introducing thunks for all bindings, essentially switching to an implementation of lazy evaluation.
As for a compiler check: This could always only be approximative, right? I don't think there is a way to do this reliably (catching all cases of recursive dependencies in values) without being too conservative (i.e., catching all said cases but still permitting all definitions that in reality can be evaluated without ending up in a nonterminating situation).

The (first) point being: If you need this (lazy) behavior, then in Elm you will have to introduce the laziness explicitly, not expect the language to behave that way automatically. Introducing the desired laziness can be done in a domain specific API (which is why a added the recursively combinator to the parser library), or it can be done using the more generic approach of http://package.elm-lang.org/packages/maxsnew/lazy/latest. (Of course, the latter generic approach can be used for implementing the former for specific APIs.)

I don't think that this form of recursive value requires laziness though. This doesn't really require recursion so much as cycles. This kind of thing is commonly done in two pass form:
var a = {};
var b = {};
a.siblings = [b];
b.siblings = [a];

The existing compiler could be made to do this by having an optional last parameter generated for constructors which passes in the object to construct on top of, and turning the one phase output into a two stage output. I'm sure this is a vast oversimplification of the work involved; but in theory this should work without any form of laziness.
I think the compiler check should always be exact and computable without sacrificing much. Cycle detection in the AST should work I imagine. You'd have to mark nodes as initialization-time (constants, direct references) and post-init time ( thunks, functions, etc) and only need to check the init-time ones. You do lose some cases though -- in the snippet I posted, if you flip the order of declaration it works, and if you did this cycle check the compiler should always throw you an error on it. To me though, this is desirable -- it is unintuitive behavior when order of declarations matters sometimes and not others.
I think, despite work involved (of which I'll be happy to help out on if this kind of thing is deemed appropriate to the language), this type of thing should be allowed. My concern here is that I don't entirely see a simple way to work with graphs without it, and that would somewhat limiting. Manually using the lazy functionality would be a work around to force a sort of two phase initialization, but this is also forcing the user to be aware of implementation details in the compiler.
(Which isn't to say there aren't other valid use cases for the lazy package -- I just don't think it is the best approach to this usage).

Is this a duplicate of #873? If so, check out the full description of how things work in OCaml that is linked there and see if it actually addresses your case. The restriction that you must use constructors explicitly seems to make it hard to use in a lot of cases.

This is the same issue; apologies for not finding it via search.
The functionality that is indicated in OCaml is pretty much what I want/expected here. Namely --

For this class of expressions, the actual value of x is not needed to
evaluate E: a pointer to x suffices.  Evaluation proceeds by binding x
to a dummy object, then evaluating E, then overwriting x with the
value of E.

Is this a feasible thing implementation wise / is this something you think elm should have capabilities for?

@jdavidberger, I don't understand your proposed implementation strategy in general, but maybe it is not too far off to say that it would indeed by some kind of lazy evaluation, specialized to a very specific situation? But independently of that question (whether it is/requires lazy evaluation), I question whether it would be a good idea to allow this in Elm.
Elm is, in all other respects, a strictly evaluated language. So if there are definitions
a = f b
b = g a

in a program, one should expect evaluating either a or b to not terminate. (I do agree that it is unfortunate the current implementation terminates for one, but not for the other. But I think the fix should be such that both are treated in the same strict manner, namely not terminate.)
If I understand correctly, you are proposing that for certain choices of f and g in the above snippet, Elm should behave in a guaranteed terminating fashion. Or would your approach work for arbitrary terminating f and g, not just for the case that f and g are syntactically built just from constructors? If a restriction to constructors would be in place, that would be a threat to referential transpareny: f might be defined somehow recursively but ultimately evaluate to just a constructor. Then the implementation would behave differently depending on whether I directly write the constructor there or whether I write f (which is semantically equivalent to that constructor after reduction)?
Also, considering just the case that f and g are indeed just constructors, say something like:
a = 1 :: b
b = 2 :: a

If this is compiled to something terminating, putting a and b in memory as cons cells with cross-references, then I would very much expect the following to also work:
a' () = 1 :: b' ()
b' () = 2 :: a' ()

In Haskell, these two code snippets are semantically equivalent, in the sense that you can replace a by a' () in any client of this module without any difference in observable program output (or in termination). The same should be the case in Elm, actually in any pure functional programming language (because, again, otherwise the language loses referential transparency). But as I understand, under your proposal these two code snippets would not be semantically equivalent.
So actually I think that after your proposal is implemented, the programmer would have to be aware of compiler implementation details in order to understand what the program means (namely that for the first code snippet some magic happens, but not for the second).

I don't understand your proposed implementation strategy in general, but maybe it is not too far off to say that it would indeed by some kind of lazy evaluation, specialized to a very specific situation? But independently of that question (whether it is/requires lazy evaluation), I question whether it would be a good idea to allow this in Elm.

The linked discussion -- https://groups.google.com/forum/m/#!msg/comp.lang.ml/4eK7ucslWRw/CDc9S5AGYSYJ -- does a much better and more formal job of explaining what I was after. Some recursive expressions have a fixed point solution, and some work goes into finding out which ones do and then representing them.
In terms of the example you gave, I'm not sure I understand what wouldn't be semantically equivalent about those two expressions. As it currently exists, the later versions would hang, but doesn't this have more to do with no built-in memoization? This goes back to the
ones = 1 :: ones

example as well. This actually could work in either form with generated code something like
ones = {}
ones.head = 1;
ones.tail = ones;

where you could perform some list operations without hitting a stack limit. I'm not sure infinite lists like this should be considered fixed point for elms purposes though; so I would agree that for these examples; in either case, an error (hopefully compile-time) should be generated.

Yeah, I've looked at the email describing how (some) ML does it as well now. I don't like it. :-)
And the reason is exactly that it depends on the syntax of the expression. If expression E is formed so-and-so (conservative assumptions: using only constructors and other constants, but no function calls), then the trick will work. But If I write another expression E', which is semantically equivalent to  E, but happens to not fulfill the certain set of syntactic restrictions, then the trick does not anymore work. This is what I mean by not-referentially-transparent. Might be a matter of taste, and my expectations about referential transparency as a Haskell progammer.
About the discussion of a vs. a' (): What I meant is that "in my Haskell world" both head a and head (a' ()) would terminate and evaluate to 1, and in "my Elm world" both head a and head (a' ()) would hang, while in "your Elm world" head a would evaluate to 1 but head (a' ()) would hang. Or wouldn't it?
Likewise, I think that even after the "ML trick" is implemented,
ones () = 1 :: ones ()

would mean that head (ones ()) hangs. Do you agree with this? (Your statement that ones would work "in either form" suggests something else, but maybe I am misunderstanding.)

Okay, I'm going to close as a duplicate then :)
I personally think the syntactic thing that OCaml is restrictive enough to be kind of useless. In any case, part of Joey's work on dead code elimination will make it quite easy to detect safe recursive values and allow them when the recursion is guarded by a lambda. I don't know if we will go farther than that. Let's revisit this once the initial restrictive implementation is released.

I see what you are saying about seemingly equivalent expressions giving differing results. And I'd agree that this isn't ideal. Even with very good error message support it could hard to determine exactly what is blocking compilation so that it could be resolved.
In terms of making (a' ()) equivalent to a: With memoization support I think it does what you expect, without it it would always hang. I think that is a large argument for memoizing these forms of expressions if this kind of data structure is included in the language. Without memoization head (ones ()) hangs; with it, it should return 1.
In practical terms, I'm not that bothered by equivalent expressions having differing termination behaviors. Ideally everything would terminate with the 'right' answer, but sometimes there is no fixed-point right answer (sum ones)[Actually I guess Infinity would be correct, but there are other series which addition diverges on] and in other cases there is, but most compilers won't/shouldn't try to find it (sum zeroes), even though one does exist.
I think we can both agree though that if both forms of the expression do terminate, they must have the same value, or chaos ensues. :)
@evancz Sounds good, thanks for looking into this!

Yes, what I was writing about referential transparency was mainly a matter of principle. Pragmatically, one might make some compromises. (Also, Evan, absolutely okay to close this here, of course. Am just continuing the exchange with David (?) here for the intellectual fun of it.)
So, memoization, yes that would work for ones () = .... (I wasn't getting you meant general memoization, not the specific form of it that is part of implementing lazy evaluation. So I was a bit confused.) But, for the sake of discussing this to the end, what about the following?
ones n = 1 :: ones (n+1)

Since ones is not really strict in n, any call like ones 42 should still be equivalent to the ones from ones = 1 :: ones. But this time, memoization will not help to make this happen.

@JoeyEremondi, this issue specifically is the one that I that may be made worse by #1024. Currently it only seems to apply to functions defined in the same module, whereas the proposed output code #1024 will have to topologically sort function definitions across modules.

@zeckalpha, we have it under control.
The new format cannot make things worse because modules must form a directed acyclic graph (DAG) meaning that there can be no mutually recursive values across modules.
The issue about giving proper warnings on dangerously recursive values is a separate topic. We have noticed that it seems closely related, but the actual code is pretty orthogonal either way. Please trust us to sort it out, we think about these things and would very much like to resolve the core issue here.


[Material Components] OverlayWindow lacks frame constructor

Sorry I'm not opening a PR for this, I'm still very new to Xamarin and totally clueless on bindings. OverlayWindow inherits from UIWindow, but doesn't seem to expose the frame constructor. I was expecting to be able to new up an instance like this:
new MaterialComponents.OverlayWindow(UIScreen.MainScreen.Bounds);

VS bug #805946

@J-Swift  i think we have released at least one update to material components.  Can you check in the latest version and if this is still an issue re log it using the new issue templates


No way of knowing what run options are being applied

Describe the bug
There is no indication if various run_option kwargs are actually being applied.  Eg. all of these pass through and return results:
from qiskit_ibm_runtime import Sampler
with Sampler(circuits=[ReferenceCircuits.bell()]*3, service=service,
             options={ "backend": backend.name}) as sampler:
    result = sampler(circuit_indices=[0, 1, 2], transpile_config={'optimization_level': 3})
    print(result)

from qiskit_ibm_runtime import Sampler
with Sampler(circuits=[ReferenceCircuits.bell()]*3, service=service,
             options={ "backend": backend.name}) as sampler:
    result = sampler(circuit_indices=[0, 1, 2], transpile_config={'hello_there': 3})
    print(result)

from qiskit_ibm_runtime import Sampler
with Sampler(circuits=[ReferenceCircuits.bell()]*3, service=service,
             options={ "backend": backend.name}) as sampler:
    result = sampler(circuit_indices=[0, 1, 2], transpiler_config={'is-it-transpile_or_transpiler': 3})
    print(result)

Steps to reproduce
Expected behavior
The program should error on invalid options.
Suggested solutions
Additional Information

qiskit-ibm-runtime version:
Python version:
Operating system:

Make sure metadata is returned from program.

At the least, any run_options should be documented here https://quantum-computing.ibm.com/services?program=sampler .
Validation of input parameters that are used is implicit. When it comes to dictionaries being passed, the keys are not checked to see if they are part of a known set. I think this is fine, if the docs are clearly documenting all the parameters that would work.

Is this, #243 and #240 duplicates of each other?

There is really nothing in the services description that tell me what the valid options are:

And because there is no validation of inputs, nor any confirmation that my settings sticks, it is not clear if I am setting things correctly.

Moving this issue to ntc-ibm-programs so estimator.json and sampler.json can be updated
Will update the relevant docs in this repo in issue #240