[Draft] 0.22.0 Release Notes

Apache Druid 0.22.0 contains over 400 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 73 contributors. See the complete set of changes for additional details.
# New features
# Query engine
# Support for multiple distinct aggregators in same query
Druid now can support multiple DISTINCT 'exact' counts using the grouping aggregator typically used with grouping sets. Note that this only applies to exact counts - when druid.sql.planner.useApproximateCountDistinct is false, and can be enabled by setting druid.sql.planner.useGroupingSetForExactDistinct to true.
https://github.com/apache/druid/pull/11014
# SQL ARRAY_AGG and STRING_AGG aggregator functions
The ARRAY_AGG aggregation function has been added, to allow accumulating values or distinct values of a column into a single array result. This release also adds STRING_AGG, which is similar to ARRAY_AGG, except it joins the array values into a single string with a supplied 'delimiter' and it ignores null values. Both of these functions accept a maximum size parameter to control maximum result size, and will fail if this value is exceeded. See SQL documentation for additional details.
https://github.com/apache/druid/pull/11157
https://github.com/apache/druid/pull/11241
# Bitwise math function expressions and aggregators
Several new SQL functions functions for performing 'bitwise' math (along with corresponding native expressions), including BITWISE_AND, BITWISE_OR, BITWISE_XOR and so on. Additionally, aggregation functions BIT_AND, BIT_OR, and BIT_XOR have been added to accumulate values in a column with the corresponding bitwise function. For complete details see SQL documentation.
https://github.com/apache/druid/pull/10605
https://github.com/apache/druid/pull/10823
https://github.com/apache/druid/pull/11280
# Human readable number format functions
Three new SQL and native expression number format functions have been added in Druid 0.22.0, HUMAN_READABLE_BINARY_BYTE_FORMAT, HUMAN_READABLE_DECIMAL_BYTE_FORMAT, and HUMAN_READABLE_DECIMAL_FORMAT, which allow transforming results into a more friendly consumption format for query results. For more information see SQL documentation.
https://github.com/apache/druid/issues/10584
https://github.com/apache/druid/pull/10635
# Expression aggregator
Druid 0.22.0 adds a new 'native' JSON query expression aggregator function, that lets you use Druid native expressions to perform "fold" (alternatively known as "reduce") operations to accumulate some value on any number of input columns. This adds significant flexibility to what can be done in a Druid aggregator, similar in a lot of ways to what was possible with the Javascript aggregator, but in a much safer, sandboxed manner.
Expressions now being able to perform a "fold" on input columns also really rounds out the abilities of native expressions in addition to the previously possible "map" (expression virtual columns), "filter" (expression filters) and post-transform (expression post-aggregators) functions.
Since this uses expressions, performance is not yet optimal, and it is not directly documented yet, but it is the underlying technology behind the SQL ARRAY_AGG, STRING_AGG, and bitwise aggregator functions also added in this release.
https://github.com/apache/druid/pull/11104
# SQL query routing improvements
Druid 0.22 adds some new facilities to provide extension writers with enhanced control over how queries are routed between Druid routers and brokers. The first adds a new manual broker selection strategy to the Druid router, which allows a query to manually specify which Druid brokers a query should be sent to based on a query context parameter brokerService to any broker pool defined in druid.router.tierToBrokerMap (this corresponds to the 'service name' of the broker set, druid.service).
The second new feature allows the Druid router to parse and examine SQL queries so that broker selection strategies can also function for SQL queries. This can be enabled by setting druid.router.sql.enable to true. This does not affect JDBC queries, which use a different mechanism to facilitate "sticky" connections to a single broker.
https://github.com/apache/druid/pull/11566
https://github.com/apache/druid/pull/11495
# Avatica protobuf JDBC Support
Druid now supports using Avatica Protobuf JDBC connections, such as for use with the Avatica Golang Driver, and has a separate endpoint from the JSON JDBC uri.
String url = "jdbc:avatica:remote:url=http://localhost:8082/druid/v2/sql/avatica-protobuf/;serialization=protobuf";

https://github.com/apache/druid/pull/10543
# Improved query error logging
Query exceptions have been changed from WARN level to ERROR level to include additional information in the logs to help troubleshoot query failures. Additionally, a new query context flag, enableQueryDebugging has been added that will include stack traces in these query error logs, to provide even more information without the need to enable logs at the DEBUG level.
https://github.com/apache/druid/pull/11519
# Streaming Ingestion
# Task autoscaling for Kafka and Kinesis streaming ingestion
Druid 0.22.0 now offers experimental support for dynamic Kafka and Kinesis task scaling. The included strategies are driven by periodic measurement of stream lag (which is based on message count for Kafka, and difference of age between the message iterator and the oldest message for Kinesis), and will adjust the number of tasks based on the amount of 'lag' and several configuration parameters. See Kafka and Kinesis documentation for complete information.
https://github.com/apache/druid/pull/10524
https://github.com/apache/druid/pull/10985
# Avro and Protobuf streaming InputFormat and Confluent Schema Registry Support
Druid streaming ingestion now has support for Avro and Protobuf in the updated InputFormat specification format, which replaces the deprecated firehose/parser specification used by legacy Druid streaming formats. Alongside this, comes support for obtaining schemas for these formats from Confluent Schema Registry. See data formats documentation for further information.
https://github.com/apache/druid/pull/11040
https://github.com/apache/druid/pull/11018
https://github.com/apache/druid/pull/10314
https://github.com/apache/druid/pull/10839
# Kafka ingestion support for specifying group.id
Druid Kafka streaming ingestion now optionally supports specifying group.id on the connections Druid tasks make to the Kafka brokers. This is useful for accessing clusters which require this be set as part of authorization, and can be specified in the consumerProperties section of the Kafka supervisor spec. See Kafka ingestion documentation for more details.
https://github.com/apache/druid/pull/11147
# Native Batch Ingestion
# Support for using deep storage for intermediary shuffle data
Druid native 'perfect rollup' 2-phase ingestion tasks now support using deep storage as a shuffle location, as an alternative to local disks on middle-managers or indexers. To use this feature, set druid.processing.intermediaryData.storage.type to deepstore, which uses the configured deep storage type.
Note - With "deepstore" type, data is stored in shuffle-data directory under the configured deep storage path, auto clean up for this directory is not supported yet. One can setup cloud storage lifecycle rules for auto clean up of data at shuffle-data prefix location.
https://github.com/apache/druid/pull/11507
# Improved native batch ingestion task memory usage
Druid native batch ingestion has received a new configuration option, druid.indexer.task.batchProcessingMode which introduces two new operating modes that should allow batch ingestion to operate with a smaller and more predictable heap memory usage footprint. The CLOSED_SEGMENTS_SINKS mode is the most aggressive, and should have the smallest memory footprint, and works by eliminating in memory tracking and mmap of intermediary segments produced during segment creation, but isn't super well tested at this point so considered experimental. CLOSED_SEGMENTS, which is the new default option, eliminates mmap of intermediary segments, but still tracks the entire set of segments in heap, though it is relatively well tested at this point and considered stable. OPEN_SEGMENTS will use the previous ingestion path, which is shared with streaming ingestion and performs a mmap on intermediary segments and builds a timeline so that these segments can be queryable by realtime queries. This is not needed at all for batch, but OPEN_SEGMENTS mode can be selected if any problems occur with the 2 newer modes.
https://github.com/apache/druid/pull/11123
https://github.com/apache/druid/pull/11294
https://github.com/apache/druid/pull/11536
# Allow batch tasks to wait until segment handoff before completion
Druid native batch ingestion tasks can now be optionally configured to not terminate until after the ingested segments are completely loaded by Historical servers. This can be useful for scenarios when the trade-off of keeping an extra task slot occupied is worth using the task state as a measure of if ingestion is complete and segments are available to query.
This can be enabled by adding awaitSegmentAvailabilityTimeoutMillis to the tuningConfig in the ingestion spec, which specifies the maximum amount of time that a task will wait for segments to be loaded before terminating. If not all segments become available by the time this timeout expires, the job will still succeed. However, in the ingestion report, segmentAvailabilityConfirmed will be false. This indicates that handoff was not successful and these newly indexed segments may not all be available for query. On the other hand, if all segments become available for query on the Historical services before the timeout expires, the value for that key in the report will be true.
This tuningConfig value is not supported for compaction tasks at this time. If a user tries to specify a value for awaitSegmentAvailabilityTimeoutMillis for Compaction, the task will fail telling the user it is not supported.
https://github.com/apache/druid/pull/10676
# Data lifecycle management
# Support managing segment and query granularity for auto-compaction
Druid manual and automatic compaction can now be configured to change segment and query granularity. Additionally, compaction will preserve segment granularity by default. This allows operators to more easily perform options like changing older data to larger segment and query granularities in exchange for decreased data size. See  compaction docs for details.
https://github.com/apache/druid/pull/10843
https://github.com/apache/druid/pull/10856
https://github.com/apache/druid/pull/10900
https://github.com/apache/druid/pull/10912
https://github.com/apache/druid/pull/11009
# Allow compaction to temporarily skip locked intervals
Druid auto-compaction will now by default temporarily skip locked intervals instead of waiting for the lock to become free, which should improve the rate at which datasources can be compacted. This is controlled by druid.coordinator.compaction.skipLockedIntervals, and can be set to false if this behavior is not desired for some reason.
https://github.com/apache/druid/pull/11190
# Support for additional automatic metadata cleanup
You can configure automated cleanup to remove records from the metadata store after you delete delete some entities from Druid:

segments records
audit records
supervisor records
rule records
compaction configuration records
datasource records created by supervisors

This feature helps maintain performance when you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources or other related entities. You can limit the length of time to retain unused metadata records to prevent your metadata store from filling up. See automatic cleanup documentation for more information.
https://github.com/apache/druid/pull/11078
https://github.com/apache/druid/pull/11084
https://github.com/apache/druid/pull/11164
https://github.com/apache/druid/pull/11200
https://github.com/apache/druid/pull/11227
https://github.com/apache/druid/pull/11232
https://github.com/apache/druid/pull/11245
# Dropping data
A new setting, dropExisting has been added to the ioConfig of Druid native batch ingestion tasks and compaction, which if set to true (and appendToExist is false), then the ingestion task will transactionally mark all existing segments in the interval as unused, replacing them with the new set of segments. This can be useful in compaction use cases where normal overshadowing does completely replace a set of segments in an interval, such as when changing segment granularity to a smaller size and some of the smaller granularity buckets would have no data, leaving the original segments only partially overshadowed.
Note that this functionality is still experimental,  and can result in temporary data unavailability for data within the compacted interval. Changing this config does not cause intervals to be compacted again.
Similarly, markAsUnused has been added as an option to the Druid kill task, which will mark any segments in the supplied interval as 'unused' prior to deleting all of the unused segments. This is useful for allowing the mark unused -> delete sequence to happen with a single API call for the caller, as well as allowing the unmark action to occur under a task interval lock.
https://github.com/apache/druid/pull/11070
https://github.com/apache/druid/pull/11025
https://github.com/apache/druid/pull/11501
# Coordinator
# Control over coordinator segment load timeout timeout behavior with Apache Zookeeper based segment management
A new Druid coordinator dynamic configuration option allows controlling the behavior whenever a segment load action times out when using Zookeeper based segment management. replicateAfterLoadTimeout when set to true, the coordinator will attempt to replicate the segment that failed to load to a different historical server. This helps improve the segment availability if there are a few slow historical servers in the cluster. However, the slow historical may still load the segment later and the coordinator may need to issue drop requests if the segment is over-replicated.
https://github.com/apache/druid/pull/10213
# Faster coordinator segment balancing
Another new coordinator dynamic configuration option, useBatchedSegmentSampler, when set to true can potentially provide a large performance increase in the speed which the coordinator can process the segment balancing phase. This should be particularly notable at very large cluster sizes with many segments, but is disabled by default to err on the side of caution.
https://github.com/apache/druid/pull/11257
# Improved loadstatus API to optionally compute under-replication based on cluster size
The Druid coordinator load status API now supports a new optional URL query parameter, computeUsingClusterView, which when specified will cause the coordinator compute under-replication for segments based on the number of servers available within cluster that the segment can be replicated to, instead of the configured replication count configured in load rule. For example, if the load rules specify 2 replicas, but there is only 1 server which can hold segments, this API would not report as under-replicated because the segments are as replicated as is possible for the given cluster size.
https://github.com/apache/druid/pull/11056
# Optional limits on the number of non-primary replicants loaded per coordination cycle
A new coordinator dynamic configuration, maxNonPrimaryReplicantsToLoad, with default value of Integer.MAX_VALUE, lets operators to define a hard upper limit on the number of non-primary replicants that will be loaded in a single coordinator execution cycle. The default value will mimic the behavior that exists today.
Example usage: If you set this configuration to 1000, the coordinator will load a maximum of 1000 non-primary replicants in each run cycle execution. Meaning if you ingested 2000 segments with a replication factor of 2, the coordinator would load 2000 primary replicants and 1000 non-primary replicants on the first execution. Then the next execution, the last 1000 non-primary replicants will be loaded.
https://github.com/apache/druid/pull/11135
# Web Console
# General improvements
The Druid web-console 'services' tab will now display which coordinator and overlord servers are serving as the leader, displayed in the 'Detail' column of the table. This should help operators be able to more quickly determine which node is the leader and thus which likely has the interesting logs to examine.
The web-console now also supports using ASCII control characters, by entering them in the form of \uNNNN where NNNN is the unicode code point for the character.
https://github.com/apache/druid/pull/10951
https://github.com/apache/druid/pull/10795
# Query view
The query view of the web-console has received a number of 'quality of life' improvements in Druid 0.22.0. First, the query view now provides an indicator of how long a query took to execute:

Also, queries will no longer auto-run when opening a fresh page, to prevent stale queries from being executed when opening a browser, the page will be reset to 0 if the query result changes and the query limit will automatically increase when the last page is loaded re-running the query.
Inline documentation now also should include Druid type information:

and should provide better suggestions whenever a query error occurs:

Finally, the web console query view now supports the hot-key combination command + enter (on mac) and ctrl + enter on Windows and Linux.
https://github.com/apache/druid/pull/11158
https://github.com/apache/druid/pull/11128
https://github.com/apache/druid/pull/11203
https://github.com/apache/druid/pull/11365
# Data management
The web-console segments view timeline now has the ability to pick any time interval, instead of just the previous year!

The web-console segments view has also been improved to hopefully be more performant when interacting with the sys.segments table, including providing the ability to 'force' the web-console to only use the native JSON API methods to display segment information:

The lookup view has also been improved, so that now 'poll period' and 'summary' are available as columns in the list view:

We have also added validation for poll period to prevent user error, and improved error reporting:

https://github.com/apache/druid/pull/11359
https://github.com/apache/druid/pull/10909
https://github.com/apache/druid/pull/11620
# Metrics
# Prometheus metric emitter
A new "contrib" extension has been added, prometheus-emitter, which allows Druid metrics to be sent directly to a Prometheus server. See the extension documentation page for complete details: https://druid.staged.apache.org/docs/0.22.0/development/extensions-contrib/prometheus.html
https://github.com/apache/druid/pull/10412
https://github.com/apache/druid/pull/11618
# ingest/notices/queueSize
ingest/notices/queueSize is a new metric added to provide monitoring for supervisor ingestion task control message processing queue sizes, to help in determining if a supervisor might be overloaded by a large volume of these notices. This metric is emitted by default for every running supervisor.
https://github.com/apache/druid/pull/11417
# query/segments/count
query/segments/count is a new metric which has been added to track the number of segments which participate in a query. This metric is not enabled by default, so must be enabled via a custom extension to override which QueryMetrics are emitted similar to other query metrics that are not emitted by default. (We know this is definitely not friendly, and hope someday in the future to make this easier, sorry).
https://github.com/apache/druid/pull/11394
# Cloud integrations
# AWS Web Identity / IRSA Support
Druid 0.22.0 adds AWS Web Identity Token Support, which allows for the use of IAM roles for service accounts on Kubernetes, if configured as the AWS credentials provider.
https://github.com/apache/druid/pull/10541
# S3 ingestion support for assuming a role
Druid native batch ingestion from S3 input sources can now use the AssumeRole capability in AWS for cross-account file access. This can be utilized by setting assumeRoleArn and assumeRoleExternalId on the S3 input source specification in a batch ingestion task. See AWS documentation and native batch documentation for more details.
https://github.com/apache/druid/pull/10995
# Google Cloud Storage support for URI lookups
Druid lookups now support loading via Google Cloud Storage, similar to existing functionality available with S3. This requires the druid-google-extensions must be loaded in addition to the lookup extensions, but beyond that it is as simple as using a Google Cloud Storage URI.
https://github.com/apache/druid/pull/11026
# Other changes
# Extracting Avro union fields by type
Avro ingestion using Druid batch or streaming ingestion now supports an alternative mechanism of extracting data for Avro Union types. This new option, extractUnionsByType only works when utilizing a flattenSpec to extract nested data from union types, and will cause the extracted data to be available with the type as part of the flatten path. For example, given a multi-typed union column someMultiMemberUnion, with this option enabled a long value would be extracted by $.someMultiMemberUnion.long instead of $.someMultiMemberUnion, and would only extract long values from the union. See Avro documentation for complete information.
https://github.com/apache/druid/pull/10505
# Support using MariaDb connector with MySQL extensions
Druid MySQL extensions now supports using the MariaDB connector library as an alternative to the MySQL connector. This can be done by setting druid.metadata.mysql.driver.driverClassName to org.mariadb.jdbc.Driver and includes full support for JDBC URI parameter whitelists used by JDBC lookups and SQL based ingestion.
https://github.com/apache/druid/pull/11402
# Add Environment Variable DynamicConfigProvider
Druid now provides a DynamicConfigProvider implementation that is backed by environment variables. For example:
druid.some.config.dynamicConfigProvider={"type": "environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}}

See dynamic config provider documentation for further information.
https://github.com/apache/druid/pull/11377
# Add DynamicConfigProvider for Schema Registry
Ingestion formats which support Confluent Schema Registry now support supplying these parameters via a DynamicConfigProvider which is the newer alternative to PasswordProvider. This will allow ingestion tasks to use the config provider to supply this information instead of directly in the JSON specifications, allowing the potential for more secure manners of supplying credentials and other sensitive configuration information. See data format and dynamic config provider documentation for more details.
https://github.com/apache/druid/pull/11362
# Security fixes
# Control of allowed protocols for HTTP and HDFS input sources
Druid 0.22.0 adds new facilities to control the set of allowed protocols used by HTTP and HDFS input sources in batch ingestion. druid.ingestion.hdfs.allowedProtocols is configured by default to accept hdfs as the protocol, and druid.ingestion.http.allowedProtocols by default will allow http and https. This might cause issue with existing deployments since it is more restrictive than the current default behavior in older versions of Druid, but overall allows operators more flexibility in securing these input sources.
https://github.com/apache/druid/pull/10830
# Fix expiration logic for LDAP internal credential cache
This version of Druid also fixes a flaw in druid-basic-security extension when using LDAP, where the credentials cache would not correctly expire, potentially holding expired credential information after it should have expired, until another trigger was hit or the service was restarted. Druid clusters using LDAP for authorization should update to 0.22.0 whenever possible to fix this issue.
https://github.com/apache/druid/pull/11395
# Ingestion supervisor and task APIs now all consistently require WRITE access to DATASOURCE resources
0.22.0 fixes some inconsistent permissions on several ingestion APIs which would incorrectly allow users with READ access to get details on supervisor specs and other ingestion task information. This has been fixed to require WRITE access, which is required to perform ingestion.
https://github.com/apache/druid/pull/11680
# Performance improvements
# General performance

improved granularity processing speed: https://github.com/apache/druid/pull/10904
improved string comparison performance: https://github.com/apache/druid/pull/11171

# JOIN query enhacements

improved performance for certain JOIN queries by allowing some INNER JOIN queries to be translated into native Druid filters: https://github.com/apache/druid/pull/11068
support filter pushdown into the left base table for certain JOIN queries, controlled by new query context parameter enableJoinLeftTableScanDirect (default to false): https://github.com/apache/druid/pull/10697

# SQL

improved SQL group by query performance by using native query granularity when possible: https://github.com/apache/druid/pull/11379
added druid.sql.avatica.minRowsPerFrame broker configuration which can be used to significantly improve JDBC performance by increasing the result batch size: https://github.com/apache/druid/pull/10880
faster SQL parsing through improved expression parsing and exception handling: https://github.com/apache/druid/pull/11041
improved query performance for sys.segments: https://github.com/apache/druid/pull/11008
reduced SQL schema lock contention on brokers: https://github.com/apache/druid/pull/11457
improved performance of segmentMetadata queries which are used to build SQL schema https://github.com/apache/druid/pull/10892

# Vectorized query engine

vectorized query engine support for DataSketches quantiles aggregator: https://github.com/apache/druid/pull/11183
vectorized query engine support for DataSketches theta sketch aggregator: https://github.com/apache/druid/pull/10767
vectorized query engine support for Druid cardinality aggregator: https://github.com/apache/druid/pull/11182
vectorization support has been added for expression filter: https://github.com/apache/druid/pull/10613
vectorized group by support for string expressions: https://github.com/apache/druid/pull/11010
deferred string expression evaluation support for vectorized group by engine: https://github.com/apache/druid/pull/11213
improved column scan speed for LONG columns with 'auto' encoding (not the default): https://github.com/apache/druid/pull/11004
improved column scan/filtering speeds when contiguous blocks of values are read: https://github.com/apache/druid/pull/11039

# Bug fixes
Druid 0.22.0 contains over 80 bug fixes, you can see the complete list here.
# Upgrading to 0.22.0
Consider the following changes and updates when upgrading from Druid 0.21.x to 0.22.0. If you're updating from an earlier version than 0.21.0, see the release notes of the relevant intermediate versions.
# Dropped support for Apache ZooKeeper 3.4
Following up to 0.21, which officially deprecated support for Zookeeper 3.4, which has been end-of-life for a while, support for ZooKeeper 3.4 is now removed in 0.22.0. Be sure to upgrade your Zookeeper cluster prior to upgrading your Druid cluster to 0.22.0.
https://github.com/apache/druid/issues/10780
https://github.com/apache/druid/pull/11073
# Native batch ingestion segment allocation fix
Druid 0.22.0 includes an important bug-fix in native batch indexing where transient failures of indexing sub-tasks can result in non-contiguous partitions in the result segments, which will never become queryable due to logic which checks for the 'complete' set. This issue has been resolved in the latest version of Druid, but required a change in the protocol which batch tasks use to allocate segments, and this change can cause issues during rolling downgrades if you decide to roll back from Druid 0.22.0 to an earlier version.
To avoid task failure during a rolling-downgrade, set
druid.indexer.task.default.context={ "useLineageBasedSegmentAllocation" : false }

in the overlord runtime properties, and wait for all tasks which have useLineageBasedSegmentAllocation set to true to complete before initiating the downgrade. After these tasks have all completed the downgrade shouldn't have any further issue and the setting can be removed from the overlord configuration (recommended, as you will want this setting enabled if you are running Druid 0.22.0 or newer).
https://github.com/apache/druid/pull/11189
# SQL timeseries no longer skip empty buckets with all granularity
Prior to Druid 0.22, an SQL group by query which is using a single universal grouping key (e.g. only aggregators) such as SELECT COUNT(*), SUM(x) FROM y WHERE z = 'someval' would produce an empty result set instead of [0, null] that might be expected from this query matching no results. This was because underneath this would plan into a timeseries query with 'ALL' granularity, and skipEmptyBuckets set to true in the query context. This latter option caused the results of such a query to return no results, as there are no buckets with values to aggregate and so they are skipped, making an empty result set instead of a 'nil' result set. This behavior has been changed to behave in line with other SQL implementations, but the previous behavior can be obtained by explicitly setting skipEmptyBuckets on the query context.
https://github.com/apache/druid/pull/11188
# Druid reingestion incompatible changes
Batch tasks using a 'Druid' input source to reingest segment data will no longer accept the 'dimensions' and 'metrics' sections of their task spec, and now will internally use a new columns filter to specify which columns from the original segment should be retained. Additionally, timestampSpec is no longer ignored, allowing the __time column to be modified or replaced with a different column. These changes additionally fix a bug where transformed columns would be ignored and unavailable on the new segments.
https://github.com/apache/druid/pull/10267
# Druid web-console no longer supports IE11 and other older browsers
Some things might still work, but it is no longer officially supported so that newer Javascript features can be used to develop the web-console.
https://github.com/apache/druid/pull/11357
# Changed default maximum segment loading queue size
Druid coordinator maxSegmentsInNodeLoadingQueue dynamic configuration has been changed from unlimited (0) to instead to 100. This should make the coordinator behave in a much more relaxed manner during periods of cluster volatility, such as a rolling upgrade, but caps the total number of segments that will be loaded in any given coordinator cycle to 100 per server, which can slow down the speed at which a completely stopped cluster is started and loaded from deep storage.
https://github.com/apache/druid/pull/11540
# Developer notices
# CacheKeyBuilder moved from druid-processing to druid-core
The CacheKeyBuilder class, which is annotated with @PublicAPI has been moved from druid-processing to druid-core so that expressions can extend the Cacheable interface to allow expressions to generate cache keys which depend on some external state, such as lookup version.
https://github.com/apache/druid/pull/11358
# Query engine now uses new QueryProcessingPool instead of ExecutorService directly
This impacts a handful of method signatures in the query processing engine, such as QueryRunnerFactory and QuerySegmentWalker to allow extensions to hook into various parts of the query processing pool and alternative processing pool scheduling strategies in the future.
https://github.com/apache/druid/pull/11382
# SegmentLoader is now extensible and customizable
This allows extensions to provide alternative segment loading implementations to customize how Druid segments are loaded from deep storage and made available to the query engine. This should be considered an unstable api, and is annotated as such in the code.
https://github.com/apache/druid/pull/11398
# Known issues
For a full list of open issues, please see https://github.com/apache/druid/labels/Bug.
# Credits
Thanks to everyone who contributed to this release!
@2bethere
@a2l007
@abhishekagarwal87
@AKarbas
@AlexanderSaydakov
@ArvinZheng
@asdf2014
@astrohsy
@bananaaggle
@benkrug
@bergmt2000
@camteasdale143
@capistrant
@Caroline1000
@chenyuzhi459
@clintropolis
@cryptoe
@DaegiKim
@dependabot[bot]
@dkoepke
@dongjoon-hyun
@egor-ryashin
@fhan688
@FrankChen021
@gianm
@harinirajendran
@himadrisingh
@himanshug
@hqx871
@imply-jbalik
@imply-jhan
@isandeep41
@jasonk000
@jbampton
@jerryleooo
@jgoz
@jihoonson
@jon-wei
@josephglanville
@jp707049
@junegunn
@kaijianding
@kazuhirokomoda
@kfaraz
@lkm
@loquisgon
@MakDon
@maytasm
@misqos
@mprashanthsagar
@mSitkovets
@paul-rogers
@petermarshallio
@pjain1
@rohangarg
@samarthjain
@shankeerthan-kasilingam
@spinatelli
@sthetland
@suneet-s
@techdocsmith
@Tiaaa
@tushar-1728
@viatcheslavmogilevsky
@viongpanzi
@vogievetsky
@vtlim
@wjhypo
@wx930910
@xvrl
@yuanlihan
@zachjsh
@zhangyue19921010

Druid manual and automatic compaction can now be configured to change segment and query granularity.

The linked docs and the linked PRs make it look like only manual compaction supports changing query granularity.  Either this sentence is wrong, or the docs should be updated (and more PRs should be linked). cc @maytasm who did most of the linked PRs.

The linked docs and the linked PRs make it look like only manual compaction supports changing query granularity. Either this sentence is wrong, or the docs should be updated (and more PRs should be linked). cc @maytasm who did most of the linked PRs.

Good catch, this was my mistake when doing the writeup :+1:


Compile Times

Compile times aren't completely horrible, but they do seem currently to be a little on the high side for e.g. Turing.jl (@yebai if you have a particular example in mind, please feel free to provide it below).
No effort has yet been made to make compile times good, so there is probably a lot of low-hanging fruit here for someone who knows what they're doing.

Thanks for this Hong.
Re PrecompileTools.jl -- we'll have to do some work to figure out if this is the right approach. It's a really effective tool if there's just a handful of functions that you need to precompile, but at the minute I suspect that we're over-specialising in a lot of places, which would make using PrecompileTools.jl etc less effective. We'll need to first figure out where most of the time is spent, and the figure out what the right solution is.


Do we need to precompute mutation likelihoods for sample edges?

I have a feeling that we only once need to use the likelihood of a sample edge (e.g. the array of probs from a sample node u at time 0 to its parent at all older timeslices). Is that right? In which case can we skip over the calc in get_mut_ll() and just calculate the array when we need it for that node?

I believe that’s true… and it might in fact be true for every edge? That’s the way I used to have the code structured, I took the get_mut_ll calculation out to try and simplify the forward algorithm, but it would be better to calculate the likelihood at every step in the forward algorithm to save on memory

On Dec 6, 2019, at 10:03 PM, Yan Wong<EMAIL_ADDRESS>wrote:
I have a feeling that we only once need to use the likelihood of a sample edge (e.g. the array of probs from a sample node u at time 0 to its parent at all older timeslices). Is that right? In which case can we skip over the calc in get_mut_ll() and just calculate the array when we need it for that node?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/awohns/tsdate/issues/74?email_source=notifications&email_token=ACRLT2GNRBTQ6TGUGYBPRWTQXLD25A5CNFSM4JXCHST2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H6YOHNA, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRLT2AJJTRSW72TSGPD6IDQXLD25ANCNFSM4JXCHSTQ.

Closing, as https://github.com/awohns/tsdate/pull/81 doesn't cache sample edges. Other edges possibly are worth saving in a cache, as their spans will be shorter, so are less likely to be unique


oci: wasm modules are no longer stored in the wasm architecture

The wasm program is now in an OCI layer.
Before this patch, the OCI image had 3 architectures with only one layer each:

amd64
arm64
wasm

After this patch, the OCI image has 2 architectures:

amd64
arm64

And each architecture has 1 or 2 layers with different media types:

application/vnd.gadget.ebpf.program.v1+binary
application/vnd.gadget.wasm.program.v1+binary

This patch initially comes from #2204. But I would like to get it merged early before the rest of #2204 is ready.
This is a potentially breaking change. However, the currently published image-based gadgets don't make use of wasm yet, so no existing gadget shoud break.

This PR is a prerequisite for the rest of the work on:

#2202 (#2204)
#2045

The CI still fails on cri-o despite the PR being rebased after #2367.

The CI still fails on cri-o despite the PR being rebased after #2367.

Looking at the failed logs:
It appears that the expected event is there, but it is not enriched. It has no runtime information, nor k8s information. Other events like the one from the nginx-pod are correctly enriched

Thanks for checking. This seems unrelated to this PR, so could I merge it?

Thanks for checking. This seems unrelated to this PR, so could I merge it?

Go for it!


Update V8: 12.6 -> 12.7

This PR updates V8 from 12.6 to 12.7. My only concern here is the TracedReference dereferencing API removal. For now, I've just reverted the removal, and things seem to work -- but we need to find a way to move off of it soon.

Based on https://chromium-review.googlesource.com/c/v8/v8/+/5600069 and https://chromium-review.googlesource.com/c/v8/v8/+/5596550 we should also update FP16 and abseil here to keep their versions in sync with V8.

Thanks @fhanau, I added an Abseil and FP16 update. I also updated to V8 <IP_ADDRESS>, which is now the latest patch release.

The Windows build was stalled due to a transient bazel issue, I kicked it and it passed. PR LGTM.


[WFCORE-2262] Latest WildFly Elytron and Invocation Changes

:bangbang: Commits from this branch are used in numerous other topic branches so please do not cherry-pick or rebase. :bangbang:
The following issues are resolved by this PR: -
https://issues.jboss.org/browse/WFCORE-1984
https://issues.jboss.org/browse/WFCORE-2078
https://issues.jboss.org/browse/WFCORE-2115
https://issues.jboss.org/browse/WFCORE-2239
https://issues.jboss.org/browse/WFCORE-1891
https://issues.jboss.org/browse/WFCORE-2261
https://issues.jboss.org/browse/WFCORE-2262
https://issues.jboss.org/browse/WFCORE-2263
WildFly PR https://github.com/wildfly/wildfly/pull/9595
Experimental Builds: -
Linux https://ci.wildfly.org/viewLog.html?buildId=44066&
Windows https://ci.wildfly.org/viewLog.html?buildId=44068&
Linux experimental build with disabled tests enabled: -
https://ci.wildfly.org/viewLog.html?buildId=44067&
(Expect approximately 91 failures)

Core - Full Integration Build 4295 outcome was FAILURE using a merge of e341b879b85263cbde87493e3e50f5d0798e85a7
Summary: Tests passed: 84; compilation error: Compiler (new) Build time: 00:01:19

Full integration - Windows Build 2716 outcome was FAILURE using a merge of e341b879b85263cbde87493e3e50f5d0798e85a7
Summary: Tests passed: 84; compilation error: Compiler (new) Build time: 00:05:15

New windows experimental build https://ci.wildfly.org/viewLog.html?buildId=44095&

#2149 is this plus a commit that addresses the deprecation issue I mentioned above. Feel free to pull that commit into your branch, or if the deprecation wasn't wanted and the text is just cruft ignore it, or this can go in as is and mine can go in later, or @kabir can merge #2149 instead of this one. Kabir, any of these are fine with me, so consider this PR to have my +1.

@bstansberry FYI this PR is from a repository within the wildfly-security-incubator organisation so pull requests can be submitted directly against that PR - I will merge yours over there.

Latest experimental builds: -
https://ci.wildfly.org/viewLog.html?buildId=44113&
https://ci.wildfly.org/viewLog.html?buildId=44114&

@darranl thanks Darran; I'll close the other one. I sent it up here so if Kabir was dealing with getting this + the full one in while you were away he could just do it without messing with wildfly-security-incubator.  But it's cleaner now, having this all together.

A new experimental build: -
https://ci.wildfly.org/viewLog.html?buildId=44121&
https://ci.wildfly.org/viewLog.html?buildId=44122&


On very skinny screens, photo and descriptions should stack

On very skinny screens, e.g. a smartphone, the photo and descriptive text should stack instead of sitting side-by-side. Here's an example from Chicago's site:

http://servicetracker.cityofchicago.org/requests/13-01221149

Well, this was fixed in PR #79. Closing.


WIP: Cleanup on aisle travis

Hooray! Looks like datamuxer is python 3.5 compatible too.  This PR should be good to go also.

Much cleaner. Thanks.


Prometheus server high CPU usage for Label sorting

Using prometheus as monitor on kubernetes cluster. ~100 node_exporters, ~300 targets total. Running go tool pprof --gif $server/debug/pprof/profile > prof.gif shows high CPU usage for sorting Labels.

We're running prometheus as a fairly by-the-books kubernetes custom-metrics-apiserver, using k8s-prometheus-adapter.
The code (https://github.com/prometheus/prometheus/blob/master/pkg/labels/labels.go) seems to have two reasons for running slowly: one is passing through Sort interface, the other is the repeated comparison of strings with a possibly shared prefix. Using labels to represent paths in a hierarchy seems like a reasonable use case, and shouldn't lead to poor performance. If anyone has a one-liner to generate the list of labels being operated on, I can dump them.
If my diagnosis is correct, there are low-hanging solutions. One is to go:generate a sort procedure which does not pass through the interface barrier (several generators are floating around). This is simplest, but does not address the shared-prefix problem. Two is to implement a trie-sort, which does.
System information:
Centos 7.4
Linux 3.10.0-693.11.6.el7.x86_64 x86_64
Prometheus version:
2.1.0
Prometheus configuration file:
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-apiservers
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: default;kubernetes;https
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-nodes
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - replacement: kubernetes.default.svc:443
    target_label: __address__
  - regex: (.+)
    replacement: /api/v1/nodes/${1}/proxy/metrics
    source_labels:
    - __meta_kubernetes_node_name
    target_label: __metrics_path__
  scheme: https
  tls_config:

Can you try with a newer version of Prometheus? You're probably hitting an issue with SD that was fixed.

Just bumped to 2.2.0. Will post Monday. Thanks for the quick response!


Dropdown options are hiding down, if there is no room space.

Normal dropdowns are displaying the options over the debugging tools,but the ui-select dropdown is displaying the options under the debugging tools.
Not able to select the options in this scenario.

+1 for drop above. Default browser select controls do it, anything that wishes to replace that functionality should also do it!
I'm a ui-select2 user who is aching to be able to switch to ui-select, and this is definitely a requirement.

+1

+1

I think this is already resolved in a version. Added a screenshot what i have.

@user378230 , still a enhancement (maybe i'm missing something here) or can we close it?

Not sure, this is a old issue though, so we'll close it. I'm sure someone will shout if they have a problem still.

This is still an issue, unless I missed a fix for it somewhere along the way. In my case I have a selectmenu within a modal dialog.

I have the same issue, when I want to select, it goes behind my next item.

And heres the code:

	      Completar los datos para estacionar

		    Calle seleccionada

		    {{estacionar.calle | json}}

		    Calle

			  <ui-select ng-model="estacionar.calle" theme="selectize" ng-disabled="disabled" style="width: 300px;" title="Calles">
			      <ui-select-match placeholder="Escribir calle...">{{$select.selected.name}}</ui-select-match>
			      <ui-select-choices repeat="calle in calles | filter: $select.search">

			      </ui-select-choices>
			  </ui-select>


Suppress non-parenthesis warnings

Which issue(s) this PR fixes:
Fixes #
What this PR does / why we need it:
This patch will suppress following warnings:
$ ruby -w -I"lib:test" test/plugin/test_out_http.rb
test/plugin/test_out_http.rb:445: warning: ambiguity between regexp and two divisions: wrap regexp in parentheses or add a space after `/' operator
test/plugin/test_out_http.rb:446: warning: ambiguity between regexp and two divisions: wrap regexp in parentheses or add a space after `/' operator
test/plugin/test_out_http.rb:482: warning: ambiguity between regexp and two divisions: wrap regexp in parentheses or add a space after `/' operator
test/plugin/test_out_http.rb:483: warning: ambiguity between regexp and two divisions: wrap regexp in parentheses or add a space after `/' operator

Seems we will see this warning If we omit the parentheses in the method and pass a regular expression as an argument.
irb(main):001> $VERBOSE=true
=> true
irb(main):002> p /m/
(irb):2: warning: ambiguity between regexp and two divisions: wrap regexp in parentheses or add a space after `/' operator
/m/
=> /m/

Docs Changes:
Release Note:

These tests are for sigv4 feature, which is added in v1.17.0:

#4590

So, we don't need backport this.


Depression Task: Approach 1: tf_idf speed up process

Testing different approaches to speed up the process in order to decrease run time so I can process all sentences

Also doing the same for vectorizing the sentence into one vector and getting there cosin similarities as well


Add autopep8 GitHub Action

Description
This pull request adds a GitHub Action workflow to automatically run autopep8 on pull requests. The workflow is configured to format the codebase using autopep8 and ensure that all new code adheres to our coding standards before merging.
The workflow file .github/workflows/autopep8.yml has been created, which:

Checks out the repository
Sets up Python
Installs autopep8
Runs autopep8 on the entire codebase
Checks for any uncommitted changes after running autopep8 and fails the workflow if changes are detected

Motivation and Context
I saw that the project already encourages the manual use of autopep8 before submitting pull requests. This change enforces code formatting consistency automatically by running the formatter for you. Automating this process reduces the likelihood of formatting issues slipping through code reviews and simplifies the development workflow.

By integrating this automated workflow, we can maintain a clean and consistent codebase more efficiently, allowing developers to focus more on functionality and less on formatting.
Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[x] I have added an explanation of what your changes do and why you'd like us to include them.
[ ] I have written new tests for your core changes, as applicable.
[ ] I have successfully ran tests with your changes locally.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

Looks like I misunderstood the use of this tool. It actually does not push any code itself, but just checks to make sure that the code is all formatted already. In this case the repo will need to be formatted manually the first time.

@garlic-os it might be worth checking out ruff, which includes a linter and formatter. It also allows you to only check new code going forward, so you don't have to reformat the whole codebase:
https://docs.astral.sh/ruff/tutorial/#adding-rules
We also provide a re-usable ruff action here:
https://github.com/ASFHyP3/actions/#reusable-ruffyml

I followed the instructions at ASFHyP3/actions/#reusable-ruffyml and put it in place of the autopep8 runner. @jhkennedy I trust you have some familiarity with it, would you know how to set it up to only check new code like you mentioned?

@garlic-os it looks I didn't quite remember right; you can have ruff add noqa statements for things that currently flag but that will also end up with a PR that has a lot of changes. So it's probably best to fix what you can easily in a PR and then noqa the rest.
Alternatively, you can use a tool like darker:
https://github.com/astral-sh/ruff/discussions/4049#discussioncomment-8026767
which will allow just the changes to be checked, but you won't be able to use the ASFHyP3/actions reusable action other than as a starting point for a new action.

No, sorry. I meant to convert it back to a draft. I feel #672 should happen before this one now.


feat: adding local provisioning with static info

Fixes Issue #8
Initially adding the static pools.

Codecov Report

Merging #14 (7005856) into main (f93f592) will decrease coverage by 0.67%.
The diff coverage is 78.29%.

@@             Coverage Diff              @@
##               main      #14      +/-   ##
============================================
- Coverage     87.24%   86.56%   -0.68%
- Complexity      301      325      +24
============================================
  Files            53       54       +1
  Lines          1466     1593     +127
  Branches         39       50      +11
============================================
+ Hits           1279     1379     +100
- Misses          167      183      +16
- Partials         20       31      +11

Impacted Files
Coverage Δ

...ilcali/device/pool/model/ProvisionOutputModel.java
100.00% <ø> (ø)

...ce/pool/client/DeviceLabProvisionServiceModel.java
84.84% <ø> (-0.23%)
:arrow_down:

...ice/pool/provision/LocalProvisionServiceModel.java
78.12% <78.12%> (ø)

...n/java/me/philcali/device/pool/BaseDevicePool.java
91.30% <100.00%> (ø)

.../java/me/philcali/device/pool/BaseDeviceModel.java
100.00% <0.00%> (+5.88%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f93f592...7005856. Read the comment docs.


Survey Results graph shows error when there are no results

Login as any role
Go to home page
expand an open engagement
Check the last graph

Private Zenhub Image
looking good


[Bug]: benchmark random input-len inconsistent

Your current environment
vllm/vllm-openai:latest
Model Input Dumps
No response
🐛 Describe the bug
I test benchmark script and set the --random-input-len=8000 for Meta-Llama-3.1-8B-Instruct model，the server api shows that it gets more than 8000 input
the input param

the server log

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

and the total output token is not random_output_len*num_prompts？


Different output when build with opt and dbg for boxtracker

build for the same graph with opt and dbg option but get different output.
build with opt:
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/object_tracking:object_tracking_cpu
build with dbg:
bazel build -c dbg --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/object_tracking:object_tracking_cpu
run the compiled binary for the same video:
bazel-bin/mediapipe/examples/desktop/object_tracking/object_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/tracking/object_detection_tracking_desktop_live.pbtxt --input_video_path=/media/data/palm_detection/test_videos/IMG_2430_3.MOV
visualization:

build with dbg:

build with opt:

Hi @laosanlyu, Could you please share the video to test the above scenario and let us know if there are any code changes w.r.t the above. Thanks!

Hi @laosanlyu, Could you please share the video to test the above scenario and let us know if there are any code changes w.r.t the above. Thanks!
Sorry, the video file is too big to upload, there is no code changes. any videos will get similar result

closed by mistake

Hello @laosanlyu,
We are ending support for these MediaPipe Legacy Solutions, but upgrading the others. However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.
You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions.


Create a "new project" command

It'd be nice if we could easily extract a kelpie theme into an existing folder/git repo, so lets add an easy command to do that 😄

Here's what it would optimally do:

Download a copy of Kelpie from this repo using the "Clone" button.
Remove any *.lock files.
Rename any files with "kelpie" in the name to your theme's namespace. (This should be the main theme folder, plus the classes files in /classes.)
Run a find-and-replace to get all instances of "Kelpie" and "kelpie" (case-sensitive!) with your theme name.
Find-and-replace 'octopusthink' with your organisation's GitHub organisation.


OpenIDConnect error 500

I get this error after authenticating.
This page isn’t working
localhost is currently unable to handle this request.
HTTP ERROR 500
It seems to me the OpenIDConnect Middleware is not handling the the providers POST to "https://localhost:44359/signin-oidc"
I thought the middleware is supposed to handle this endpoint automatically.
I am using Microsoft.AspNetCore.Authentication.OpenIdConnect 2.0.0

500 usually means an exception was thrown while processing the request. Run it under the debugger or enable logging to see the exception.


When connection disappears the retries option is not taken into account

When connection disappears or if never was on the first place then the NoKafkaConnectionError is thrown.
ERROR no-kafka-client Metadata request failed: NoKafkaConnectionError [<IP_ADDRESS>:9092]: Error: connect ECONNREFUSED
The retries.delay and retries.attempts are not used in that case. The default 1000 value is used instead.
https://github.com/oleksiyk/kafka/blob/master/lib/client.js#L155
Can submit a PR which uses the delay provided in the Producer options. Otherwise, please fix. Thanks!

Producer.send retries option has no relation to the Client.metadataRequest (Client instance is also shared by GroupConsumer, GroupAdmin and etc). By design Client.metadataRequest has an endless loop trying to reach initial brokers. It can't use any limited number of attempts because otherwise the library will loose its state (more important for consumers, not for producers).

Can it be customizable at least?

reconnectionDelay option is available in 2.5.0: https://github.com/oleksiyk/kafka/commit/76285190e535c7da51f71c98f1203efcab3527d1

Amazing news! Thank you Oleksiy.

Producer._send calls updateMetadataRequest() on retry and this goes into infinite loop when NoKafkaConnectionError is thrown from client.js:produceRequest.
So, effectively, producer never gets an error on a send, when retries > 1.
Is there any way for the producer.send to return an error on Kafka errors instead of infinite looping?

@bviswana101 there is no way to get out of reconnection loop.

@oleksiyk Can reconnectionDelay option be set for the producer.init() ?

@agouz  reconnectionDelay is a library option, it applies to both producer and consumer.

@oleksiyk Thanks for replying. How can I set it so the producer.init() pick it up? currently it runs in an endless loop coz of invalid kafka url.

I think all options are well documented in a README:
var producer = new Kafka.Producer({
    reconnectionDelay: {
        min: 1000,
        max: 10000
    }
});

FYI: i got around the endless loop issue, by using bluebird Promise.timeout() option.

It would be great to have an event emitter which would fire when connection
established or lost.
On 14 Oct. 2017 02:42, "bviswana101"<EMAIL_ADDRESS>wrote:

FYI: i got around the endless loop issue, by using bluebird
Promise.timeout() option.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/oleksiyk/kafka/issues/64#issuecomment-336489818, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABjCL56t6whK-sN1IDPeeqEGhztZ6kB7ks5sr4TrgaJpZM4If6es
.


Add PSA labels in the namespace created by Template

Added the PSA labels in the template as the latest OC 4.12 has the PSA by default created to restricted mode.

@gauravAggrahari Can you please review and approve the PR so that we can get unblocked.

@gauravAggrahari since we have the approvals can we merge this now?

@asifdxtreme can you please test this Template and past e the results ?


how to visualizate the semseg result?

I run the module and get the acc, iou result, but I want to visualizate the semseg result, how can i do it?

see #8

The visualization code is added in this repo.


ploidy setting for mtDNA & -w parameter syntax

Hi!
I was wondering what the correct ploidy setting would be for mtDNA. Can it be freely scaled? So if we expect >2 variant bases at a position, can we set --ploidy it to 3 or 4?
Moreover, is the -w parameter strictly necessary and what is the correct syntax? The help (-h) says ctg:start-end, while in https://github.com/jts/nanopolish/issues/181 it is chr:start-end.
When I run
qf0297@qf0297-ESPRIMO-P420:/media/qf0297/Volume/data_np/20170627_mtDNA_HEK/alba/fast5_alba$ /home/qf0297/nanopolish/nanopolish variants --reads mtdna_reads_alba.fasta --bam sorted_mtdna_alba.bam --genome rCRS_ME.fasta --ploidy 1 -w ctg:1-16569 > mtdna_snps_ploidy1.vcf
it reports
nanopolish: src/alignment/nanopolish_alignment_db.cpp:364: void AlignmentDB::load_region(const string&, int, int): Assertion `m_region_end >= 0' failed.
Aborted (core dumped)

Also using "chr" instead of "ctg" gives the same error.
No error occurs if i omit -w . The reference is exactly 16569 bases long: so is it correct to say chr:1-16569?
Thank you
Stefan

Hi,
nanopolish uses a discrete genotype model so it can't be freely scaled. It doesn't work well for continuous allele frequencies, like you would have in a population.
When making the -w option you should replace "chr" with the name of the chromosome (or contig) you want to process. For example if your chromosome name is "chrA" you would write -w chrA:1000-2000.
I hope this helps.
Jared

I have clarified the -w usage in 609f6fd


Re-enable Search Input in DataTable

After trying to use the DataTable component with the new Material UI Implementation, the DataTable wasn't rendered at all  get the following error and
Uncaught (in promise) Invariant Violation: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.

After some debugging, I determine the error was related with the following component inside DataTable render
{
  showSearch &&

  <Components.SearchInput value={this.state.query}
                          updateQuery={this.updateQuery}
                          className={classes.search}
  />
}

Now, this is how looks in my Datatable.

Also, I fix some issues related to ErrorCatcher components, which is part of Vulcan errors packages, I added that package as a dependency.
Thanks to @digitalhydra who help me to do the re-implementation.

Nice! You also updated the ModalTrigger?

@eric-burel yes but I think was only because we were testing some other issues and our editor "fix" the indentation. Should I remove that?
I am checking some issues with <Components.ErrorCatcher> that don't allow the Modal to render the form.

Also, I fix some issues related to ErrorCatcher components, which is part of Vulcan errors packages, I added that package as a dependency.
let me know if I have to split into two PR.

@eric-burel I'll let you merge this one and all Material UI stuff generally since I don't use it myself.

@SachaG Yep no problem I'll handle them
@enzolutions It's ok to keep one PR, just double-check that it does not contain unexpected changes, and indeed try to cancel them if possible.
We are close to get Storybook to work (it works for bootstrap but I just need to figure out what needs to be specifically imported for Material UI) but until then it's a bit tedious to test the UI, so I'll trust you on those changes and test them later. Just ping me when you're done so I can review and merge.

And again thanks for the help, this is very welcome !

Perfect.
I will try to review the select issue.

Nice fixes, it already solves a few issues.
I've noticed that error messages are not correctly displayed, I'll check way.
As you've already noticed there's a small change in the onChange function API. I don't know if it comes from Vulcan or Material UI itself.

let me know if I can help you with the error messages.


add AuthPlugin as vue plugin

i think it would improve usage of AuthPlugin when we add the instance to vue by vue plugin install.
this would allow usage of  some public methods / properties directly in every vue instance:
some methods could be used very often vue components e.g. this.$auth.authenticated or this.$auth.hasRole

it might be useful to separate the current AuthPlugin and a possible vue plugin to allow continued use in non vue projects. A separate vue plugin could import the AuthPlugin and install it to the vue instance, while still allowing the Plugin to be used outside of vue.

not planned for now, as this would also add vuejs depenceny and currently we use it also in non-vue projects.


How to use it?

I think this is what we really need. But the question is simple. How to use it? Any step by step to integrated with PURECSS.io?
Regards
Jauhari

Hi there. I want to try this on a static site I'm putting together. I'm still a bit green in many areas of development. Can you explain how to implement your files into mine?

Hi @mpwashburn! Thanks for reaching out. Happy to help you out.
I would start by downloading the source (https://github.com/pimbrouwers/YUI-Pure-CSS-Extensions/archive/master.zip).
Once you unarchive it, there will be a folder insider called dist. In there is a single file. I would copy that file into your project folder and reference it in the <head></head> of your document using the <link rel="stylesheet" type="text/css" href=""></link> tag. Note that you'll need to update the HREF to be the relative path to your copy of the css file.
Hopefully that helps.

@jauhari Documentation has been added to the readme. Thanks for reaching out!


The requested URL returned error: 522

Using Paper and a few Spigot mods, installed with the env vars.
Everything worked great, until I just tried adding two new mods to the env var, and it now repeatedly errors on startup.
Removing the two new mods from the env var has no effect, so I'm now wondering if it's failing to update/check for updates to Paper itself?
Any idea what causes the following?
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxrwx. 9 1000 1000 4096 May 25 06:30 /data'
[init] Resolved version given LATEST into 1.16.5
[init] Resolving type given PAPER
curl: (22) The requested URL returned error: 522
curl: (22) The requested URL returned error: 522
[init] WARN: using  since that's the latest provided by PaperMC
curl: (22) The requested URL returned error: 522
curl: (22) The requested URL returned error: 522
[init] WARN: using  since that's the latest provided by PaperMC
curl: (22) The requested URL returned error: 522
curl: (22) The requested URL returned error: 522
[init] WARN: using  since that's the latest provided by PaperMC
curl: (22) The requested URL returned error: 522
curl: (22) The requested URL returned error: 522
[init] WARN: using  since that's the latest provided by PaperMC

I'd add as a second workaround option in the meantime that the PAPER_DOWNLOAD_URL environment variable enabled me to get back up and running.
In my case, I just pointed it at the link shown on the PaperMC website's download page for getting their latest version. I don't know if it is the latest stable build that they put there though. But for the only container that I rebooted and got stuck in that situation, it wasn't much of a concern.

Thanks for the workaround idea @campfred
I'll see if I can at least improve the error handling since the loop of messages is not good
curl: (22) The requested URL returned error: 522
curl: (22) The requested URL returned error: 522
[init] WARN: using  since that's the latest provided by PaperMC
curl: (22) The requested URL returned error: 522
curl: (22) The requested URL returned error: 522

You're welcome, @itzg!
I'll add that for the error messages, my logs were showing a 404 error instead, which is what threw me off because I couldn't believe that PaperMC would vanish from the face off heart on a pinch (even though it is possible). So, if you're looking into reacting to specific error codes, you might want to take the 404 one in account too.


Create Raymond arizona recommendation file

This is a bullet point.

[ ] This is a task list item.

General Comment


add adapter api and explicit version

To solve some of the challenges we've had in keeping adapters up to date with respect to changes in the juttle runtime, the API used by adapter implementations should be both more clearly defined and explicitly versioned.
Currently adapters require various modules from inside the juttle project. This means that when things move around inside the repository (e.g. as happened to JuttleMoment in #312), adapters need to catch up to the changes.
Instead we should have an explicit adapter implementation API. The idea here
would be that the only way adapters would pull in portions of the juttle codebase would be via:
var AdapterAPI = require('juttle').AdapterAPI(version);
This module would compose all of the portions of the juttle compiler and runtime
that adapter implementations are allowed to use. This would include:

AdapterRead
AdapterWrite
getLogger
types => lib/runtime/types
FilterJSCompiler
reducerEmptyValue(reducer)
testUtils => juttle-test-utils

The parseTime, compileError, and runtimeError utilities would be folded
into methods on the adapter base classes.
Note that as we implement this we will likely find additional modules that
should be included.
In addition, this API should have an explicit semantic version with major and
minor components. Each adapter implementation would include the version of the
API to which it is written. This way the runtime can check if it is compatible
with the version that the implementation wants and throw a user-friendly error
when trying to load an incompatible adapter.

@go-oleg This is what we discussed the other day.
@dmajda / @bkutil  / @davidvgalbraith  / @VladVega / @mstemm What do you guys think?

that plan sounds good to me.

lgtm

Elegant! Downright professional.

For the record, we discussed this with @demmer yesterday. Main points from my side:

I totally agree with one goal of the API, which is to specify and narrow the adapter interface. While I understand that exporting an API object will make this interface more explicit than just documentation, I value regularity in the code and thus using normal requires for reaching into the juttle module.

The other goal of the API — providing versioning separate from versioning of the juttle package itself — assumes that often, there will be breaking changes in juttle which won’t affect the adapter API. This wasn’t the case so far and given our backlog it won’t be the case at least for several releases. From that POV introducing separate versioning may be premature. It is also a new concept which complicates things, so this cost must be considered vs. the cost of problems stemming from keeping just one version (possible “empty” adapter releases which just update the compatibility range).

If we go with a separate adapter API version, I’d prefer for the adapters to declare what version range they support (e.g. in package.json) rather than actively require an API object of some version from juttle. The disadvantage of this approach is that juttle can’t provide multiple API versions, but I doubt this will be a problem in practice. The advantage is that we could still use normal requires.

I assume that the main challenge with fast moving Juttle core is silent breakage/rot of adapter code - at least this is what I hit w/ influx adapter. (Maybe there are others?)
It looks that there are several approaches to addressing this:

Introduce an API
Use versioning - allow adapters to throw an error if there was an incompatible update in Juttle core or 'pin' adapter version to a specific version of juttle API - use versioning.
Testing process that makes the breakage visible. See Daria's proposal in Juttle engine

My take is that we should definitely do 1. and 3.
1 - An API will provide abstraction and make most changes safer. The interface can be also tested (e.g. for file renames, or basic instantiation). I agree with @dmajda that a simple require would be preferable to a loader.
3 - Testing against both master and the stable release will help us notice the breakage faster. See discussion below https://github.com/juttle/juttle-engine/issues/4 for details.
With regards to 2., aren't we trying to duplicate npm / jpm here? npm can already provide dependency tracking, semver checking, error reporting, etc.
So, instead of handling this in the code, I think it might be possible to create a npm package juttle-adapter-api, with juttle as a dependency. The package would either contain a single index.js exposing the API (via an object), or multiple files in lib, each requiring a specific part of Juttle core. It would be accompanied by a set of smoke/basic tests, and the adapters would depend on it. An advantage is independence of versions, using an existing tool with known & tested behaviour and simpler code. Also, this solution is easy to throw away and replace at any time. A disadvantage would be the cost of maintaining and releasing another package.

In part due to the spate of problems we've run into over the past couple days with npm installations being not predictable and things not finding each other properly, and taking into account feedback from @dmajda and @bkutil I'm leaning towards a modification of this proposal that removes the need for circular resolution between juttle and the adapters.
Specifically,  the Juttle runtime would export a global JuttleAdapterAPI onto which all of the above-mentioned modules would be attached. This would be the only supported way for adapters to access portions of the runtime.
The adapters would not have a peerDependency on juttle. Instead they would include a juttleAdapterVersion field in their package.json. This would be checked before loading the adapter. While I think it does make sense to have this be different from the juttle version, in practice we're likely to continue to make changes to the adapter API and other juttle internals for the near term so I agree that we should just use the same
In addition, this would enable juttle-service to directly depend on juttle rather than have it be a peerDependency (if that turned out to be helpful).
While I'm certainly sympathetic to the fact that require('juttle/path') is the standard way to reference something from a module, this isn't a standard situation. We fundamentally have a circular dependency between the juttle runtime and the adapters, which is why the prevailing model for npm simply breaks down and even looking at other systems like gulp or grunt plugins isn't a true match for what we want to do.

" Instead they would include a juttleAdapterVersion field in their package.json. This would be checked before loading the adapter. "
How would that work?

@mstemm npm allows you to put other things in package.json than the standard 'name', 'dependencies', etc.
So the idea would be that the juttle adapter core would first require() the package.json from the adapter, check that it has a compatible version, then decide to load or not to load accordingly.


python311Packages.matplotlib: 3.8.4 -> 3.9.0; add a testing derivation to passthru.tests

Motivation for this change
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

On CI:

The x86_64-darwin build timed out way before it reached matplotlib.
The x86_64-linux build shows up as "neutral", but it passed most of it so it seems, besides a weird error with installCheckPhase at the end of sage build (from passthru.tests).
The aarch64-linux build succeeded, but the passthru.tests.withOutdatedFreetype failed as expected also at https://github.com/NixOS/nixpkgs/pull/316041#issuecomment-2159216527 .
The aarch64-darwin hasn't passed yet.

So personally I think it'd be OK to let this cook in staging, but it would be nice to get an approval from the maintainers (@lovek323 and @veprbl) regarding the form of the 3.9.0 update (part of the 1st commit of the PR).


Add a teleop plugin that sends twist commands

A GUI plugin that has arrow buttons and publishes twist messages to a custom topic to control robots would be nice to test vehicles easily.
Desired behavior

The user loads a world in ign-gazebo that has a vehicle in it, like ign gazebo diff_drive.sdf.
Then the user can insert the GUI twist teleop plugin, set the desired topic, and start sending commands by pressing buttons or keys, or with sliders.
Alternatives considered

The approach currently taken for teleop is using KeyPublisher + TriggeredPublisher as described on the moving robot tutorial. This can be inconvenient because:

There's no way to load and configure TriggeredPublisher at runtime yet (https://github.com/ignitionrobotics/ign-gazebo/issues/190), so it needs to be hardcoded in the SDF file.
The combination of controls that can be made with that approach is very limited, because the speed has to be hardcoded. It isn't possible for example to increment the speed at each click.

Implementation suggestion

The rqt_robot_steering plugin for ROS has vertical and horizontal sliders to control throttle and steering.

I am tackling this on my branch: on the latest commit I added the button functionalities
https://github.com/ignitionrobotics/ign-gui/commit/31c4a4bf4bb1bfbb489a7af97b29acf9a4db70f7

You can set the desire topic and change the module of the linear and angular velocities.
For now it can only be controlled clicking on the buttons. So the next steps would be :

Allow moving the robot using keys.
Adding a switch to enable-disable the plugin.

Looks good @LolaSegura ! A suggestion would be to add a stop button. Also, it would be nice to turn while driving forward/backward (for example, driving forward and turning left at the same time)

I am tackling this on my branch

Nice! Looking good!

it would be nice to turn while driving forward/backward (for example, driving forward and turning left at the same time)

+1, that would be very helpful! I believe that's why the robot steering plugin linked above uses sliders instead of buttons.
But I think we can also achieve the same with buttons. For example, when the user presses "forward", we increment the forward velocity instead of setting it. So the more the user presses forward, the faster the robot goes. Then the same number of backward presses results in the robot slowing down until it stops. Pressing right while there's some forward velocity would make the robot go forward while rotating, etc.

Another idea is to use unicode characters with arrows on the buttons instead of the words "Forward", etc.

Looks good @LolaSegura ! A suggestion would be to add a stop button. Also, it would be nice to turn while driving forward/backward (for example, driving forward and turning left at the same time)

That's a great idea. Actually, Lola has implemented these buttons too:

This is wip so the buttons could be replaced by something else to be more user-friendly.

Another idea is to use unicode characters with arrows on the buttons instead of the words "Forward", etc.

+1000.

But I think we can also achieve the same with buttons. For example, when the user presses "forward", we increment the forward velocity instead of setting it. So the more the user presses forward, the faster the robot goes. Then the same number of backward presses results in the robot slowing down until it stops. Pressing right while there's some forward velocity would make the robot go forward while rotating, etc.

I think that approach is interesting, it is definitely good to mix forward/backward movement and clockwise rotation at the same time however I feel like probably it would be a bit confusing for the user.
What about the following:

Having buttons to move in 8 directions + stop as it is shown in the previous image. Pressing these buttons will send a twist message with the angular and linear velocity that is described in the text fields.
Allow pressing keys from the keyboard. Replicate functionality with the buttons in the visualizer
Tipically W: forward

(Sorry for the noise, I hit some key combination by mistake)

W: forward, A: left, X: backward, D: right., S: stop. (And Q, E, Z and C for the combinations)

+1 to having WASD+ keys. The advantage of keys over buttons is that the user can press more than one at the same time. So we could achieve the linear + angular combinations by combining keys instead of using new keys. For example, front + right would be W + D instead of Q.

Having buttons to move in 8 directions + stop

I'm ok with that. That's the approach in turtlebot_teleop, for example.
Another idea that came to mind is to make just 4 buttons that are togglable. So the user could enable front and right at the same time. The advantage would be that:

we have less buttons
the user knows the current state of the controller by looking at the widget and seeing which buttons are checked

Sliders

If it's not too difficult, I think they'd be a nice addition. Similar to the idea above, they allow throttle + steer combinations while providing visual feedback of the current command.

You certainly have more insight about what could be more useful to the users than me smile

Not sure about that :wink: In the end of the day, the best approach is to write something, then have someone else try to use it. In the end of the day, there's no ideal GUI control for 2D movement - that's why we have joysticks! :smile:

Another idea that came to mind is to make just 4 buttons that are togglable. So the user could enable front and right at the same time.

I implemented this idea so now there are only 5 togglable buttons. 4 for direction and 1 for stopping the vehicle. So now for turning while moving forward you have to push both 'forward' and 'right' buttons for example.
I also change the forward, left, right and backward text of the buttons for black triangles pointing to the direction. I couldn't think of any symbol for the stop button so I left it as it was. But any suggestions are welcome.
I added some visual feature so that it shows which buttons are toggled at the moment.

The advantage of keys over buttons is that the user can press more than one at the same time. So we could achieve the linear + angular combinations by combining keys instead of using new keys.

I implemented this as well. So now the keys available for moving the vehicle are W, A, D and X. I didn't include any key for stopping because the vehicle only moves while the keys are pressed.
Here is a video showing the plugin with all that I mentioned before.
Demo
The next steps to follow would be:

Adding sliders.
Solve a bug that appears when you first use the buttons and then the keyboard.

That looks great, @LolaSegura ! Feel free to open a PR once you're ready!
The last recommendation I have is to use Material Design controls so the UI looks a bit more modern. I think that changing to these imports should do the trick:
import QtQuick 2.9
import QtQuick.Controls 2.2

See for example how the Publisher plugin's TextField and Switch have a more modern look and make use of the UI's color scheme for highlights:


Allow file extensions matching when searching for layers and stylesheets

I faced this error after starting the server with a custom layer:
[ERROR] 2021-02-21 14:46:02.017 [main] Preview - Unable to read the configuration file.
com.fasterxml.jackson.databind.JsonMappingException: /config/custom/skating_mushrooms/layer.yaml (through reference chain: com.baremaps.config.Config["layers"]->java.util.ArrayList[0])
...
java.lang.NullPointerException
...

but my layer is actually named skating_mushrooms/layer.yml, therefore I suspect that the file extension is hard codded.
From here, there are probably two solutions:

baremaps should also allow for .yml extensions, e.g. within a regexp,
or it should get the full filename (including extension) so that files can be named with any extension.

The extensions are not hard coded. In your main config.yaml file, is the reference to the layer.yaml file correctly set? If the config is on github I can have a look at it.

Indeed, I forgot to rename it in the main config.yaml file. My bad.


HTTP request result is not showing with "Filter responses by environment" option

Expected Behavior
Response is shown in the Preview and Timeline area
Actual Behavior
Response area is empty
Reproduction Steps

Turn on "Filter responses by environment" option
Select an http post request
Clear History
Press "Send"

Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
Problem apperead after update to v2022.7.2
Nothing is shown in the history after I press Send with "Filter responses by environment" option = true

If I set "Filter responses by environment" option in preferences to false then everything seems working.
Insomnia Version
2022.7.2
What operating system are you using?
Windows
Operating System Version
Windows 10 Pro 22H2 build 19045.2364
Installation method
Downloaded from insomnia.rest/download some time ago and then automatically updated to 2022.7.2
Last Known Working Insomnia version
prvious version worked fine

Hi @Alterant-zz thanks for reporting this, this is a duplicate of #5664 - fix for it will go out today


Failed to scale volume 412 Precondition Failed RestrictedReplicaCount

Description
I want to scale volume replica to 1. Pod connected to volume is runnning on node6. So i expect that replica on node1 will be removed. But it is not.
Initially this volume had 1 replica on node1, I scal it to 2 replicas and then to 3 replicas and was able to scale it down to 2 replicas, but I cannot scale it to 1 replica.
kubectl mayastor get volume-replica-topology -n openebs 33c3ed9b-3684-47ad-988c-0801dcf23a62
 ID                                    NODE             POOL                 STATUS  CAPACITY  ALLOCATED  SNAPSHOTS  CHILD-STATUS  REASON  REBUILD
 ce2b1194-0d2d-4dd4-b8f9-0c7b27bf3b23  node6  pool-on-node6  Online  30GiB     13.3GiB    0 B        Online        <none>  <none>
 0a9cab43-2b09-47b9-8496-1b04fc8bdaab  node1  pool-on-node1  Online  30GiB     13.3GiB    0 B        Online        <none>  <none>

kubectl mayastor get volumes -n openebs
 ID                                    REPLICAS  TARGET-NODE      ACCESSIBILITY  STATUS  SIZE   THIN-PROVISIONED  ALLOCATED  SNAPSHOTS  SOURCE
 33c3ed9b-3684-47ad-988c-0801dcf23a62  2         node6  nvmf           Online  30GiB  true              13.3GiB    0          <none>

kubectl mayastor -n openebs scale volume 33c3ed9b-3684-47ad-988c-0801dcf23a62 1
Failed to scale volume 33c3ed9b-3684-47ad-988c-0801dcf23a62. Error error in response: status code '412 Precondition Failed', content: 'RestJsonError { details: "", message: "SvcError :: RestrictedReplicaCount: Replica Count of 1 is not attainable for AffinityGroup", kind: FailedPrecondition }'

Expected Behavior
replica on node1 removed
Current Behavior
error
Your Environment
openebs v2.7.0

RestrictedReplicaCount: Replica Count of 1 is not attainable for AffinityGroup

CC @Abhinandan-Purkait

Yeah, so this is an expected behaviour at the moment, for the stsaffinity feature.
We don't have a mechanism to ensure stsAffinity feature's consistency when you scale up and down to 1, so we restrict the operation. We have control on the scheduling while creating and not removal. So, if someone goes up in replicas when stsAffinity feature is in enabled, and then tries to come down to 1 again, it is not possible at the moment to selective remove those extra replicas while ensuring the feature still holds.


Cosmos DB sql api

It looks like a logical next step to support cosmosdb sql api query and associated repositories.

Hi @reddy6ue , yes, however we'll be only adding Cosmos DB SQL API support for URF.Core version. If you want, just recreate this issue on the https://github.com/urfnet/URF.Core repo, so we can track this request.


Memory climbs rapidly during CSG updates in example

Possible sources:

*debugger info (should possibly allow for disabling this in the demo?)
raycast hit object creation (could be fixed by #3)
list of intersection ids

With debug info disabled the memory increases much less drastically but still enough to do something about it. Raycast results and triangle index storage likely account for most of the memory allocated.

Now it seems like there's a memory leak somewhere:

Memory is now under control - other memory is heavily related to raycasting memory allocation.


Add oss and devsite domains to google search console

Google Analytics is user-oriented, providing data related to those who visit and interact with your website. Google Search Console, on the other hand, is search-engine focused, providing tools and insights that can help site owners improve visibility and presence in the SERPs.
We currently don't have our domains verified in Google Search console which we should do to get search analytics.
Google has 4 ways to validate your site We just need to decide which approach to use and implement that approach

I tried all those verification methods, and they don't validate we are looking into it and think that the Google UA tag is not being fired due to our cookie acceptance component, so Google isn't able to validate it.  @jerelmiller thinks  its because we don’t install the tracking code until after the user agrees to accept the cookies

For this we might have rewrite the way we do cookie consent in our site, so I'm going to read through the google analytic docs to see if we're able to correctly configure this.

We've tried several approaches to this and none have worked unfornately so we are going to go with the simplest which is the HTML meta tag.
@caylahamann here are the instructions, when I return I can validate this in Google Search Console. OR @zstix may have access to Search Console and he can validate it. :)
To validate you just click on the menu in the right corner of the console:

Copy the meta tag below, and paste it into your site's home page. It should go in the  section, before the first  section.

developer site

oss site

FWIW, we're verified with Google Search Console. not sure if there's anything to do except give additional peoples access?

FWIW, we're verified with Google Search Console. not sure if there's anything to do except give additional peoples access?

this is done. :)

this is done. :)


View matrix and Inverse View matrix mixed up.

Bevy version
12.0
Problem
The naming of the view and inverse_view matrices in the View struct are the opposite of what those names mean everywhere else.
A view matrix transforms from world to view space. The "view matrix" in the View struct transforms from view to world space.
https://github.com/bevyengine/bevy/blob/a4c27288c773959ff307635b4ac25399e96f5de7/crates/bevy_render/src/view/mod.rs#L381

That is an issue discussing alternative naming conventions. It does not cover the fact that there is an error in the existing names. view and inverse_view mean the opposite thing in Bevy than they do in every other renderer.

Quick summary of places with the widely accepted convention that view == world_to_camera (i.e. not Bevy's convention)
Engines

Unity: https://docs.unity3d.com/ScriptReference/Camera-worldToCameraMatrix.html
Unreal: https://docs.unrealengine.com/4.27/en-US/API/Runtime/Core/Math/FLookFromMatrix/__ctor/
Godot: https://github.com/godotengine/godot/blob/a311a4b162364d032b03ddf2a0e603ba40615ad7/drivers/gles3/shaders/scene.glsl#L412
Stride: https://github.com/stride3d/stride/blob/35bb959b83ea37518abe003c440f184e2e55ef40/sources/engine/Stride.Engine/Engine/CameraComponent.cs#L253
Flax: https://github.com/FlaxEngine/FlaxEngine/blob/526edb83de5a427daf787025e5334dead5ecd2cf/Source/Engine/Core/Math/Viewport.cs#L263

Libraries

GLM: https://github.com/g-truc/glm/blob/586a402397dd35d66d7a079049856d1e2cbab300/glm/ext/matrix_transform.hpp#L125
DirectXMath: https://github.com/microsoft/DirectXMath/blob/a4f6d7bd7897858e1bcedbd7d2127cc151b257b2/Inc/DirectXMathMatrix.inl#L2178
Apple ARKIt: https://developer.apple.com/documentation/arkit/arcamera/2921672-viewmatrix

Gfx tutorials/docs

LearnOpenGL: https://learnopengl.com/Getting-started/Camera
Vulkan Tutorial: https://vulkan-tutorial.com/Uniform_buffers/Descriptor_layout_and_buffer
DX9 Docs: https://learn.microsoft.com/en-us/windows/win32/direct3d9/view-transform
3DGEP: https://www.3dgep.com/understanding-the-view-matrix/

I could go on...
tl;dr: the convention in Bevy runs counter to the overwhelming majority of existing gfx tutorials, libraries, and engines. I can't find a single example that agrees with Bevy's convention.
IMO this is a major bug

I've also always found this odd. We should try to match the industry conversions soon than later. This is not a duplicate of https://github.com/bevyengine/bevy/issues/8492 And this issue should not be closed.

This has been discussed elsewhere. It isn't a duplicate of #8492, and I think we should resolve it by implementing #8492. I had done this, but I didn't want to disrupt @DGriffin91 's helper implementations.

I haven't thoroughly checked all of the links. Foundations of Game Engine Development 2, Rendering by Eric Lengyel uses the inverse camera matrix to transform from 'world' to 'camera' space. That uses a right-side multiplication of matrices convention. I don't remember where I originally saw the argument about "if you have a camera in space, looking into a scene, and you rotate the camera to anti-clockwise about its up axis, what happens visually to the objects the camera sees? They rotate about the camera position along the camera's up axis, but clockwise".
Not that it matters other than for trying to understand where it came from in Bevy, but I'm not certain I was the one who originally wrote this, but I feel I have understood it since, and there were definitely references.
I think to take the project forward without wasting people's time discussing whether view or inverse view is necessarily correct, we were anyway intending to do #8492 and this issue mostly suggests to me that we should either avoid referring to the 'view' matrix at all in documentation or anywhere. If we have to, we can discuss it, but maybe we could try this approach first?

This is the bit from Foundations of Game Engine Development 2.

It uses the terminology Mcamera as the matrix for the camera itself an in: camera.transform.compute_matrix(). So Mcamera is from camera space to world space.
Note though that once it switches to using view (in the form of V), V represents the inverse camera

Yes, the "camera matrix" is the transform matrix of the camera from its local space into world space.
The "view matrix" is the inverse of this matrix. This is an introductory graphics concept, and I can't find a single source that agrees with bevy's stance that "view == camera_to_world", because it's wrong!
I also think the only reason there is any confusion in your community that warrants more verbose names is because there is a massive error in the engine.
It's also not great that I've reported a bug which took me hours to track down, and the response is not to acknowledge it is bug, but to equivocate about "wasting peoples time" disccussing the issue.
At the very least, add documentation for this minor version on the wgsl and rust structs to say "hey we screwed up, this means the opposite of what it should". Such a comment would have saved me so much time.

I'd like to back up cshenton here: the question of what the "correct" function of the view matrix is unambigously answered by his list of references.
This is a unanimous agreement in graphics programming, there is no discussion about interpretation to be had.
I think the only reasonable way forward is to acknowledge that a serious bug has slipped through and to fix it quickly. Mistakes happen and that's okay.
Personally, naming the fields "world_to_camera" etc. and putting a doc on it that points out that this is also called the view matrix seems like a good choice to me, since that is more easily understandable for people less familiar with graphics programming.

I feel like the tone here has turned quite negative. I am listening and have good intentions.
When I wrote about ‘wasting time’ I was thinking how many of us maintaining the project have other full time jobs. Personally there are things I could do instead which would likely bring everyone more value. And I have seen discussions on topics like this become long, and take energy, time, and focus away from other things.
I also am ill at the moment, have seen this same discussion elsewhere, haven’t had the energy to find the origin of why it is the way it is in bevy, and I knew we were going to change the naming to be clearer anyway.
I wanted to look through the references in more detail to understand how those matrices are updated each frame in the code.
All that said, it seems there is consensus among many others. So now my proposal is the same as @ZainlessBrombie - that we do #8492 and mention that view to world is also known as the view matrix or view transformation. That way, we only break API once and address the bug.

I feel like the tone here has turned quite negative. I am listening and have good intentions.

When I wrote about ‘wasting time’ I was thinking [...]

You're right. I think there has just been a misunderstanding. Thank you, @superdump and get better soon :)

I checked git blame, and it looks like I introduced the meaning of the incorrect meaning of the camera matrix as being same as the view matrix way back when I added clustered-forward rendering to Bevy.

I've added this to the 0.13 milestone as I do think its worth resolving this in a timely fashion. Worth re-evaluating what naming conventions we want (ex: the long descriptive names vs the short "common" names).

@cart as far as I remember, we (you included) had previously decided on local, world, view, clip, and ndc for space names, and local_to_world etc for transformation names.

I highly favor the "y_from_x" convention for all matrices b/c matrix multiplication is left-to-right so when multiplying zFromY * yFromX * myVec it makes it clear that "myVec" is expected to start in X and the result will be in Z. Thus viewFromWorld and worldFromView instead of view and inverse. Space transforms are confusing enough it's worth the few extra letters.
More behind this rational here:
https://tomforsyth1000.github.io/blog.wiki.html#[[Matrix maths and names]]

This issue seemed quite quiet but had the 0.14 milestone tag on it, so I figured I'd have a go at it, primarily as food for thought. x_from_y personally feels a little weird compared to y_to_x but I can definitely see the benefit when combining transformations. Replacing x_ and inv_x (or x_inverse) with y_from_x and x_from_y is a nice win since they line up.


[tabular] Fix logging verbosity of inference throughput

Issue #, if available:
Description of changes:

Fix logging verbosity of inference throughput
It was accidentally set to 30 (verbosity>=1) instead of 15 (verbosity>=3)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Previous CI Run
Current CI Run