[Feature Request] Liquid Clustering

Overview
We propose to introduce Liquid Clustering, a new effort to revamp how clustering works in Delta, which addresses the shortcomings of Hive-style partitioning and current ZORDER clustering.
Motivation
Partitioning/clustering is a common technique to manage datasets to reduce unnecessary data processing. Hive-style partitioning and ZORDER (multi-dimensional) clustering are existing solutions in Delta, but both have limitations.
Hive-style partitioning
Hive-style partitioning clusters data such that every file contains exactly one distinct combination of partition column values. Although Hive-style partitioning works well if tuned correctly, there are limitations:

Since partition values are physical boundaries for files, Hive-style partitioning by high cardinality columns will create many small files that cannot be combined, resulting in poor scan performance.
In Spark/Delta, once a table is partitioned, its partition strategy cannot be changed, thus being unable to adapt to new use cases such as query pattern changes, etc.

ZORDER clustering
ZORDER is a muti-dimensional clustering technique used in Delta. The OPTIMIZE ZORDER BY command applies ZORDER clustering and improves the performance of queries that utilize ZORDER BY columns in their predicates. However, it has the following limitations:

Any new data ingested after the OPTIMIZE ZORDER BY run is not automatically clustered, and the user needs to rerun the command to cluster the new data.
OPTIMIZE ZORDER BY reclusters already well-clustered data, resulting in high write amplification.
ZORDER BY columns are not persisted and the user is required to remember the previous ZORDER BY columns, often causing user errors.

Solution: Liquid Clustering
This section discusses the high-level design sketch for Liquid clustering and how it fixes the problems highlighted above.
Improved space-filling curve
Liquid clustering uses a continuous fractal space filling curve as a multi-dimensional clustering technique, which significantly improves data skipping over ZORDER.
Liquid clustering can be applied to the already ingested data by running OPTIMIZE command. It improves the existing ZORDER clustering by

Incrementality: already well-clustered data is not reclustered, greatly reducing the write amplification while maintaining good data skipping. More details on the design will be shared in a separate GitHub issue or a design doc.
Ease of use: our goal is for the user to not specify ZORDER BY columns anymore, and clustering uses the current clustering columns set in the table. This is achieved by saving columns to cluster by as part of table metadata (see later).

Clustering Evolution
The user can change clustering columns based on the need (e.g., changes in query pattern) without rewriting the whole table. Once the columns for clustering are modified, incoming data is clustered using the updated clustering columns, while previously clustered data using the original clustering columns remains unaffected.
Note that we plan to allow applying new clustering columns to data that has been previously clustered using original clustering columns. This can be achieved by using the OPTIMIZE <table> RECLUSTER command (the final syntax is yet to be determined).
User surface
Liquid clustering introduces the following new SQL syntaxes:

SQL Syntax

Description

CREATE TABLE

USING delta

CLUSTER BY (<col1>, <col2>, …)

Creates a  Delta table where columns specified with CLUSTER BY are used as Liquid clustering columns.

ALTER TABLE

CLUSTER BY (<col1>, <col2>, …)

Alters the Liquid clustering columns. Data ingested after altering is clustered with new Liquid keys.

ALTER TABLE

CLUSTER BY NONE

Removes the Liquid clustering columns. No clustering is performed.

OPTIMIZE

Triggers Liquid clustering. Unlike the existing OPTIMIZE semantics - which is compacting files, OPTIMIZE on Liquid tables triggers compaction and/or clustering based on the current state of the table.

OPTIMIZE ZORDER BY is not allowed on Delta tables created with Liquid clustering columns.

Table feature / Metadata
Liquid clustering introduces a new table feature called liquid. This prevents Delta writers that are not aware of Liquid clustering from running OPTIMIZE ZORDER BY on tables with Liquid clustering.
Liquid clustering introduces a new metadata domain called delta.liquid to store clustering columns. The DomainMetadata for Liquid clustering will look like the following:

Hi guys, do we have any white paper or link that I can refer to understand MDC, or liquid clustering? I know requirement doc is on the way but any other resource that can throw more light on this feature?

This feature in on Delta Lake 3.0.0 Preview release notes, is it expected to have an 3.0.0 RC2 with the feature?
Usually, the Delta final version is released few weeks after RC1. It seems this won't make to the 3.0.0 final?

@tdas Hi，does Liquid Clustering use the DBSCAN algorithm? Thanks.

@akshaysinghUdaan: Kindly use the link, https://www.databricks.com/blog/announcing-delta-lake-30-new-universal-format-and-liquid-clustering

This needs a LOT more detail: "Liquid clustering uses a continuous fractal space filling curve as a multi-dimensional clustering technique, which significantly improves data skipping"

"For multidimensional databases, Hilbert order has been proposed to be used instead of Z order because it has better locality-preserving behavior. For example, Hilbert curves have been used to compress and accelerate R-tree indexes[9] (see Hilbert R-tree). They have also been used to help compress data warehouses.[10][11]"  [Here](https://en.wikipedia.org/wiki/Hilbert_curve#:~:text=The Hilbert curve (also known,by Giuseppe Peano in 1890.)

This needs a LOT more detail: "Liquid clustering uses a continuous fractal space filling curve as a multi-dimensional clustering technique, which significantly improves data skipping"

@loquisgon we are working on the design doc and we will have more details on clustering algorithm. Stay tuned!

@imback82 Any progress on the doc or implementing solution in delta project? Databricks 13.2 has already implemented the liquid clustering, but we still have lack of information about this solution

@vkorukanti  can you assign this issue to me?

Hello everyone, we've added the design doc to this issue. Please feel free to review it. Thank you!

@zedtang if you don't mind to share the doc using this option which allows to print and comment (if you select Commenter). Even if you leave Viewer permission only, at least that option allow us to print it.

https://support.google.com/drive/answer/2494822?hl=en&co=GENIE.Platform%3DDesktop#zippy=%2Cshare-a-file-publicly

Hi @felipepessoto , I updated the link to give everyone commenter access!

Hi, will it be possible to use liquid clustering on partitioned tables?

In the design doc I don't see any info if clustering can be applied to existing Delta tables. As we use liquid from databricks already we observed that either you have to apply liquid to an empty table or you have to use the like command which basically copies and creates a new Delta table
I guess it will be really frustrating if we can't apply liquid to existing Delta tables :)
Thanks already everyone

According to https://delta.io/pdfs/DLTDG_ER3.pdf (page 131) k-d trees are used to implement liquid clustering instead of space filling curves. Are there any pros and cons for an implementation using k-d trees?

In the design doc I don't see any info if clustering can be applied to existing Delta tables. As we use liquid from databricks already we observed that either you have to apply liquid to an empty table or you have to use the like command which basically copies and creates a new Delta table
I guess it will be really frustrating if we can't apply liquid to existing Delta tables :)

Good point. I think we should be able to migrate existing tables in place by ALTER TABLE ... CLUSTER BY .... This should be straightforward for un-partitioned tables, but it will be tricky to implement it for partitioned tables.

According to https://delta.io/pdfs/DLTDG_ER3.pdf (page 131) k-d trees are used to implement liquid clustering instead of space filling curves. Are there any pros and cons for an implementation using k-d trees?

Oh sorry about that @auckenth - this is an error within the book.  We will be updating the book chapter with the latest info per the design doc.

In the design doc I don't see any info if clustering can be applied to existing Delta tables. As we use liquid from databricks already we observed that either you have to apply liquid to an empty table or you have to use the like command which basically copies and creates a new Delta table
I guess it will be really frustrating if we can't apply liquid to existing Delta tables :)

Good point. I think we should be able to migrate existing tables in place by ALTER TABLE ... CLUSTER BY .... This should be straightforward for un-partitioned tables, but it will be tricky to implement it for partitioned tables.

@imback82 This would be great. I think considering non-partitioned tables which are optionally z-ordered is enough as a baseline. For partitioned tables you neverless accept that any changes might need an overwrite.

It'd be nice if someone could address comments and questions in the design doc. :)

Sorry if this is not the right place to ask, but is there anyway with liquid cluster to enforce some level of clustering? In particular I want to cluster by something like project,day and implement retention such that I delete data older than X days which is different for each project. With hive style I can partition on those columns and delete entire files without rewrites.

It'd be nice if someone could address comments and questions in the design doc. :)

Sorry for the delay, I replied in the design doc.

@zedtang Are you planning to support "cluster on write" ?

@zedtang Are you planning to support "cluster on write" ?

Hi @sezruby , May I ask what your use case is for this?

@zedtang With the feature, we don't need additional OPTIMIZE run after ingesting large data.
We may integrate it with optimize write feature for non-partitioned data.

It's probably tricky to implement clustering on write. (1) clustering works well only when a sufficient amount of data has accumulated and can be clustered together, so doing on every write (commonly small writes) may not produce well-clustered data at the end, and one has to run offline optimize jobs anyway (2) clustering on write will require some kind of sorting, which will come at a cost to write latency. so it's not an easy decision.
It would be very cool if someone proposes a design and shows that it will work well for some workloads without sacrificing write latency/throughput significantly

@zedtang Are you planning to support "cluster on write" ?

Hi @sezruby , May I ask what your use case is for this?

Appending large amounts of data works well, because liquid clustering is incremental. The only drawback is that you need 2x storage for appended data if you want to preserve time travel.
But if you try to merge large amounts of data (but still small compared to the rest of the table, think 200GB merged into 10TB), you have basically three options now:

not use deletion vectors, MERGE touches the majority of the files, then OPTIMIZE rewrites the majority of the files, you have just turned 10TB into 30TB
use deletion vectors, MERGE writes 200GB of new data and writes the updates as deletion vectors, then OPTIMIZE applies the deletion vectors and rewrites the majority of the files, you have just turned 10TB into 20TB
use deletion vectors, MERGE writes 200GB of new data and writes the updates as deletion vectors, then you don't run OPTIMIZE immediately, but run it once every N merges to save space

I can think of two features either of which can help with this:

have MERGE cluster the data on write if deletion vectors are enabled, so you don't have to run OPTIMIZE to cluster the data
create an additional form of OPTIMIZE that doesn't touch the deletion vectors and can be run after each merge

I am actually quite confused by the results I got right now:
I created a clustered table with 1TB or source data with deletion vectors enabled, optimized it, this doubled it in size.
spark.sql(f"""
CREATE OR REPLACE TABLE delta.`{tgt_init_table}`
USING DELTA
CLUSTER BY (eff_to_dt, eff_from_dt, id)
TBLPROPERTIES ('delta.enableDeletionVectors' = true)
AS
SELECT {', '.join(columns)}
FROM parquet.`{src_init_table}`
""")

DeltaTable.forPath(spark, tgt_init_table).optimize().executeCompaction().show(1000, False)

I vacuumed the non-optimized version. Then I enabled both write optimizations on it:
spark.sql(f"VACUUM delta.`{tgt_init_table}` RETAIN 0 HOURS")

spark.sql(f"""
ALTER TABLE delta.`{tgt_init_table}`
SET TBLPROPERTIES (
'delta.autoOptimize.optimizeWrite' = true,
'delta.autoOptimize.autoCompact' = true
)
""")

performed a MERGE of 100GB of data, which created the following three groups of files:

first some large parquet files (~ 1GB each, 100 GB in total) were written
then deletion vectors were written
then some small parquet files (~ 128MB each, 4GB in total) were written

The table history shows that it performed an OPTIMIZE right after in did a MERGE. Timestamps on the files show that the small files were written by the OPTIMIZE,
+-------+-------------------+------+--------+---------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----+--------+---------+-----------+-----------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-----------------------------------+
|version|timestamp          |userId|userName|operation                        |operationParameters                                                                                |job |notebook|clusterId|readVersion|isolationLevel   |isBlindAppend|operationMetrics                                                                                |userMetadata|engineInfo                         |
+-------+-------------------+------+--------+---------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----+--------+---------+-----------+-----------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-----------------------------------+
|6      |2024-10-28 13:01:57|NULL  |NULL    |OPTIMIZE                         |{predicate -> [], zOrderBy -> [], clusterBy -> ["eff_to_dt","eff_from_dt","id"], auto -> true}                                                                                |NULL|NULL    |NULL     |5          |SnapshotIsolation|false        |{numRemovedFiles -> 141, numRemovedBytes -><PHONE_NUMBER>56, p25FileSize -> 127702072, numDeletionVectorsRemoved -> 968, minFileSize -> 109764295, p75FileSize -> 137859299, p50FileSize -> 132611135, numAddedBytes -><PHONE_NUMBER>54, numAddedFiles -> 1371, maxFileSize -> 160050567}                                                                                |NULL        |Apache-Spark/3.5.3 Delta-Lake/3.2.1|
|5      |2024-10-28 12:43:15|NULL  |NULL    |MERGE                            |{predicate -> ["((eff_from_dt#144 = eff_from_dt#2) AND (id#142L = id#0L))"], matchedPredicates -> [{"actionType":"update"}], notMatchedPredicates -> [{"actionType":"insert"}], notMatchedBySourcePredicates -> []}                 |NULL|NULL    |NULL     |4          |Serializable     |false        |{numTargetRowsCopied -> 0, numTargetRowsDeleted -> 0, numTargetBytesRemoved -> 0, numTargetDeletionVectorsAdded -> 968, numTargetRowsMatchedUpdated -> 25002060, numTargetRowsMatchedDeleted -> 0, numTargetRowsUpdated -> 25002060, numTargetChangeFilesAdded -> 0, numTargetRowsNotMatchedBySourceDeleted -> 0, rewriteTimeMs -> 2464707, numTargetFilesAdded -> 67, numTargetBytesAdded -><PHONE_NUMBER>45, executionTimeMs -> 2959807, numTargetRowsInserted -> 74997940, numTargetDeletionVectorsUpdated -> 0, scanTimeMs -> 310651, numOutputRows -> 100000000, numTargetDeletionVectorsRemoved -> 0, numTargetRowsNotMatchedBySourceUpdated -> 0, numSourceRows -> 100000000, numTargetFilesRemoved -> 0}  |NULL        |Apache-Spark/3.5.3 Delta-Lake/3.2.1|
|4      |2024-10-28 11:52:46|NULL  |NULL    |OPTIMIZE                         |{predicate -> [], zOrderBy -> [], clusterBy -> ["eff_to_dt","eff_from_dt","id"], auto -> false}                                                                                |NULL|NULL    |NULL     |3          |SnapshotIsolation|false        |{numRemovedFiles -> 1156, numRemovedBytes -><PHONE_NUMBER>286, p25FileSize -><PHONE_NUMBER>, numDeletionVectorsRemoved -> 0, minFileSize -> 941933361, p75FileSize -><PHONE_NUMBER>, p50FileSize -><PHONE_NUMBER>, numAddedBytes -><PHONE_NUMBER>539, numAddedFiles -> 968, maxFileSize -><PHONE_NUMBER>}                                                                                |NULL        |Apache-Spark/3.5.3 Delta-Lake/3.2.1|
|3      |2024-10-28 10:10:43|NULL  |NULL    |SET TBLPROPERTIES                |{properties -> {"delta.autoOptimize.optimizeWrite":"true","delta.autoOptimize":"true","delta.autoOptimize.autoCompact":"true"}}                                                                                |NULL|NULL    |NULL     |2          |Serializable     |true         |{}                                                                                |NULL        |Apache-Spark/3.5.3 Delta-Lake/3.2.1|
|2      |2024-10-28 10:09:44|NULL  |NULL    |RESTORE                          |{version -> 0, timestamp -> NULL}                                                                                |NULL|NULL    |NULL     |1          |Serializable     |false        |{numRestoredFiles -> 1156, removedFilesSize -><PHONE_NUMBER>339, numRemovedFiles -> 1356, restoredFilesSize -><PHONE_NUMBER>286, numOfFilesAfterRestore -> 1156, tableSizeAfterRestore -><PHONE_NUMBER>286}                                                                                |NULL        |Apache-Spark/3.5.3 Delta-Lake/3.2.1|
|1      |2024-10-28 09:50:51|NULL  |NULL    |MERGE                            |{predicate -> ["((eff_from_dt#88 = eff_from_dt#2) AND (id#86L = id#0L))"], matchedPredicates -> [{"actionType":"update"}], notMatchedPredicates -> [{"actionType":"insert"}], notMatchedBySourcePredicates -> []}                   |NULL|NULL    |NULL     |0          |Serializable     |false        |{numTargetRowsCopied -> 0, numTargetRowsDeleted -> 0, numTargetBytesRemoved -> 0, numTargetDeletionVectorsAdded -> 1156, numTargetRowsMatchedUpdated -> 25002060, numTargetRowsMatchedDeleted -> 0, numTargetRowsUpdated -> 25002060, numTargetChangeFilesAdded -> 0, numTargetRowsNotMatchedBySourceDeleted -> 0, rewriteTimeMs -> 2302979, numTargetFilesAdded -> 200, numTargetBytesAdded -><PHONE_NUMBER>53, executionTimeMs -> 2786480, numTargetRowsInserted -> 74997940, numTargetDeletionVectorsUpdated -> 0, scanTimeMs -> 298298, numOutputRows -> 100000000, numTargetDeletionVectorsRemoved -> 0, numTargetRowsNotMatchedBySourceUpdated -> 0, numSourceRows -> 100000000, numTargetFilesRemoved -> 0}|NULL        |Apache-Spark/3.5.3 Delta-Lake/3.2.1|
|0      |2024-10-28 09:01:59|NULL  |NULL    |CREATE OR REPLACE TABLE AS SELECT|{partitionBy -> [], clusterBy -> ["eff_to_dt","eff_from_dt","id"], description -> NULL, isManaged -> false, properties -> {"delta.enableDeletionVectors":"true","clusteringColumns":"[[\"eff_to_dt\"],[\"eff_from_dt\"],[\"id\"]]"}}|NULL|NULL    |NULL     |NULL       |Serializable     |false        |{numFiles -> 1156, numOutputRows -><PHONE_NUMBER>, numOutputBytes -><PHONE_NUMBER>286}                                                                                |NULL        |Apache-Spark/3.5.3 Delta-Lake/3.2.1|
+-------+-------------------+------+--------+---------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----+--------+---------+-----------+-----------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-----------------------------------+

Additional OPTIMIZE commands on the table do nothing. I have no idea what happened. I'll try to roll back to version 5, VACUUM with extreme prejudice and rerun the OPTIMIZE.

I've tested OPTIMIZE + liquid clustering + deletion vectors and Delta Lake wrote much less data than I expected:
I merged 100GB of data that had about 25% of updates into 1TB table. Right now my S3 bucket looks like this:

~1TB of old data
~50GB of parquets written during MERGE, which sounds like too little data, I would expect ~100GB
~50MB of deletion vectors written during MERGE
~125GB of parquets written during OPTIMIZE

If OPTIMIZE had to rewrite every file touched by a deletion vector, given that my updates are totally random, I would expect it to write 100GB + 250GB of data, not 100GB + 25GB.
But on a smaller-sized table (100GB of old data + 10GB increment) I ended up with 220GB, just as I predicted. I am quite confused now.


Allocate colors to very-different, defining colours

The image below, for example.

The lighter coloured components in the bin are actually a rather vibrant, contrasting yellow. It'd be neat if we had the ability to pull them out as a colour.
ideas:

a depth map but for contrast. is this just edge detection? ???
pick the palette excluding some amount of allocated highly-contrasted slots. find the colours that are the furthest distance from most colours. maybe we calculate a score here like with the heuristics so we can factor in count.

related:

#6

Experimenting with an odd-factor. The Odd Factor (a la. "The Random Factor" from D20's Starstruck Odyssey) is specified as a percent of colours, rounded down, that you want to use to select from the bottom N% of the colour count.
I am not sure how we're going to do this. I think, perhaps, we'll start selecting colours when the colour we're loking at, going up from the bottom, occures at least 1% as often as the most frequent but only considering that after we're in the top 90%. So, like.
Example Conditiosn:

yellow occures 1000 times.
there are 100 unique colours.

Action

jump the first 10 colours, we're not going to consider colours from the botom 10% anyway.
keep looking at colours going up. if a colour occures at least 10 times, put it in the palette.
at this time, you can ignore those previous rules as the count list is sorted. not you're just ascending and picking as normal with the same tolerance we use for the most common.

Being able to tune those values, the bottom N% and the minimal occurrence would be cute. I want to give all the power to the user, but this also lightly complicates the code. It is worth it and makes it a more powerful tool.


Add hovertext to the job links with description of what each job does

See subject

Done in #137.


[1.8.1] After resetting a DB the new DB instance is read only

After a database is reset using FlowManager.getDatabase(Db.NAME).reset(context);, following attempts to write to the DB will throw the following exception:
java.lang.RuntimeException: android.database.sqlite.SQLiteReadOnlyDatabaseException: attempt to write a readonly database (code 1032).
I've been getting this on a Nexus 5 running 5.1. I've found this issue reported for the L-preview, but couldn't find any follow up so I'm not sure if this is still the case.

According to some posts on SO (see comments here), I believe this may be fixed by closing the DB connection before deleting the database, through SQLiteOpenHelper.close(). I've tried completely closing the app after using the current reset method and indeed the problem no longer occured.
Unfortunately the getHelper() method on BaseDatabaseDefinition isn't public, so I wasn't able to test it myself.

This still occurs in version 2.0

A little workaround for version 2.0, just to hold the helper and close the database using it:
/**
 * Sql open helper
 *
 * @author andre.moreira
 */
public class SqlOpenHelper extends FlowSQLiteOpenHelper{

	private volatile static SqlOpenHelper helper;

	/**
	 * @return The current helper created by FlowManager
	 */
	public static SqlOpenHelper getCurrent() {
		return helper;
	}

	public SqlOpenHelper(BaseDatabaseDefinition flowManager, DatabaseHelperListener listener) {
		super(flowManager, listener);
		helper = this;
	}
}

this may be outdated. If its still broken on 3.0.0-beta3 or higher, reopen new issue. Thank you


Return correct result when IDs match but types differ

From docs:

Some ID fields allow both integer and string values. When resolving references by comparing IDs, the check should fail if the IDs are different types. It should neither succeed nor N/A (it is likely to N/A if IDs are not coerced to string).

For example, awards_status should fail with this fixture. The data currently passes because the publisher was "lucky" enough to use different types for the id and awardID.
        (
            {"awards": [{"status": "pending", "id": 1}], "contracts": [{"awardID": "1"}]},
            {"processed_awards": [{"path": "awards[0]", "id": 0, "result": False}]},
            1,
            0,
        ),

Revisiting this issue:
The "main" reference check should fail if types mismatch (but it can have a useful message in the reason). Related checks should do type coercion to identify errors that might be "shadowed" by the type mismatch.

x ] reference.parties: Add different message if coerced id values match, but types don't match.
[x] consistent.roles, consistent.org_ref_name: Pass if coerced id values match. (reference.parties covers the other case)

Main check

[ ] reference.contract_in_awards

Dependent checks

[ ] coherent.awards_status
[ ] coherent.dates
[ ] consistent.contracts_value

I think I got them all now. I just did regex checks.
I think the dataset-level and time-based checks compare ocid, which should always be a string due to the prefix.


[User] Request a new tag to be approved

If a user has suggestions for new tags, they can send a request to the admin to add it. Page needs to be created.

Moving to next version of 2ofClubs :)


[SPARK-5614][SQL] Predicate pushdown through Generate.

Now in Catalyst's rules, predicates can not be pushed through "Generate" nodes. Further more, partition pruning in HiveTableScan can not be applied on those queries involves "Generate". This makes such queries very inefficient. In practice, it finds patterns like
Filter(predicate, Generate(generator, _, _, _, grandChild))

and splits the predicate into 2 parts by referencing the generated column from Generate node or not. And a new Filter will be created for those conjuncts can be pushed beneath Generate node. If nothing left for the original Filter, it will be removed.
For example, physical plan for query
select len, bk
from s_server lateral view explode(len_arr) len_table as len
where len > 5 and day = '20150102';

where 'day' is a partition column in metastore is like this in current version of Spark SQL:

Project [len, bk]
Filter ((len > "5") && "(day = "20150102")")
Generate explode(len_arr), true, false
HiveTableScan [bk, len_arr, day], (MetastoreRelation default, s_server, None), None

But theoretically the plan should be like this

Project [len, bk]
Filter (len > "5")
Generate explode(len_arr), true, false
HiveTableScan [bk, len_arr, day], (MetastoreRelation default, s_server, None), Some(day = "20150102")

Where partition pruning predicates can be pushed to HiveTableScan nodes.

Can one of the admins verify this patch?

ok to test

Test build #27055 has started for   PR 4394 at commit c03a95b.

This patch merges cleanly.

Test build #27055 has finished for   PR 4394 at commit c03a95b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27055/
Test PASSed.

Test build #27074 has started for   PR 4394 at commit c7f0003.

This patch does not merge cleanly.

Test build #27074 has finished for   PR 4394 at commit c7f0003.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27074/
Test FAILed.

Test build #27083 has started for   PR 4394 at commit 7b25c70.

This patch merges cleanly.

Test build #27086 has started for   PR 4394 at commit c7f0003.

This patch does not merge cleanly.

Test build #27086 has finished for   PR 4394 at commit c7f0003.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27086/
Test FAILed.

Test build #27090 has started for   PR 4394 at commit a67dce9.

This patch merges cleanly.

Test build #27083 has finished for   PR 4394 at commit 7b25c70.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27083/
Test PASSed.

Test build #27090 has finished for   PR 4394 at commit a67dce9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27090/
Test PASSed.

Thanks!  Merged to master and 1.3.


"python3 api\index.py" successful, but the video stream is not visible on the UI.

File "/Users/apple/Documents/2 工作空间/git/Low-Cost-Mocap-main/computer_code/api/index.py", line 55, in gen
frames = cameras.get_frames()
File "/Users/apple/Documents/2 工作空间/git/Low-Cost-Mocap-main/computer_code/api/helpers.py", line 138, in get_frames
frames = self._camera_read()
File "/Users/apple/Documents/2 工作空间/git/Low-Cost-Mocap-main/computer_code/api/helpers.py", line 72, in _camera_read
frames[i] = np.rot90(frames[i], k=self.camera_params[i]["rotation"])
ValueError: assignment destination is read-only
The source code may have issues.

That is strange. Try adding the line frame[i] = frame[i].copy() before the line that is causing the error.

The issue has been resolved, and the software can now be used by simply plugging in multiple cameras. It is recommended to have all hardware connections completed before running the software.


Blog post at 2024-12-30 disappears

Description
Today I post a new blog on our website, of which the date is 2024-12-30. But when I built the website I found that it just disappeared.
Steps To Reproduce

Find any blog post.
Changes its date to '2024-12-30'
Run hugo server

Expected Behavior
Expect as normal post
Actual Behavior
The date fails to be recognized and the post is moved to the bottom to our page
Screenshots
None, because it disappears.
Environment

Hugo Version: 0.120.3
Browser/OS: Safari, MacOS
Theme Version: latest

Additional Context
Knowing that the maintainer may be busy, we will help investigate it if possible. But may need some instructions or hint.

@xiln7 Hi, I think you need to use the --buildFuture flag to include a post with a future date.
See: https://gohugo.io/getting-started/usage/#draft-future-and-expired-content

Yes, I notice that. Thanks for your reply.


Delegate type coercion to openpyxl

Found useless))

The above commit should have referenced #341, not #342.


pam e3sm integration for both cpu and gpu

add some minor changes for integrated build in E3SM for both cpu and gpu, test build on Summit shows that these changes are necessary.

@xyuan let's close this PR, I'm pretty sure all the fixes will be included in my branch when I get around to merging it, and I think merging this might create conflicts for my branch, which includes a lot of other updates to these same files.

Sure, that sounds good!
Get Outlook for iOShttps://aka.ms/o0ukef

From: Walter Hannah @.>
Sent: Friday, June 23, 2023 8:44:45 AM
To: E3SM-Project/PAM @.>
Cc: Yuan, Xingqiu @.>; Mention @.>
Subject: Re: [E3SM-Project/PAM] pam e3sm integration for both cpu and gpu (PR #89)
@xyuanhttps://github.com/xyuan let's close this PR, I'm pretty sure all the fixes will be included in my branch when I get around to merging it, and I think merging this might create conflicts for my branch, which includes a lot of other updates to these same files.
—
Reply to this email directly, view it on GitHubhttps://github.com/E3SM-Project/PAM/pull/89#issuecomment-1604231670, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFCG2BIYMPOQM4WHXHKPYTXMWFT3ANCNFSM6AAAAAAVN2LSP4.
You are receiving this because you were mentioned.Message ID: @.***>


How to define this struct {"include": ["path/to", {"path": "path/to", "insert": "content"}]}

Here is my definition, but it seems doesn't right:
#[derive(Debug, Default, Clone, Deserialize)]
struct IncludeObj {
    path: String,
    #[serde(default = "default_insert")]
    insert: String,
}
fn default_insert() -> String {
    "use client".into()
}
#[derive(Debug, Clone, Deserialize)]
#[serde(untagged)]
enum Include {
    Str(String),
    Object(IncludeObj),
}
#[derive(Debug, Default, Clone, Deserialize)]
struct InputConfig {
    include: Vec<Include>
}

Here is the error when I run the test case:
value: Error("invalid type: string \"fixtures/include2\", expected struct IncludeConfig", line: 2, column: 33)'

Please help!🙏🙏🙏🙏

Sorry, It is my fault!


Created AI exec doc "Deploy container group to Azure virtual network"

I think this PR is outdated but this repository doesn't host the documents anymore. Closing this out for now.


housekeeping

Codecov Report

Merging #149 (cda4066) into main (3f4f677) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #149   +/-   ##
=======================================
  Coverage   69.76%   69.76%
=======================================
  Files           6        6
  Lines          86       86
  Branches       17       17
=======================================
  Hits           60       60
  Misses         23       23
  Partials        3        3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f4f677...cda4066. Read the comment docs.


Fixed the terms for ND Sens Conrad and Dorgan

Conrad switched senate seats on 12/15/1992, finishing out Burdick's term until 01/03/1995.  Dorgan took over Conrad's old senate seat on 12/15/1992, served for the remainder of Conrad's term (just a couple of weeks, from 12/15 to 01/03) and then started his full term

This is great work fixing these, but do you have a citation for this you can point to, so there's a record somewhere of why we made the change? (I would have asked for this for #426 if I'd caught it in time.)

Yeah, this is a great one. Here's what Dorgan's bioguide bio says: elected as a Democrat to the United States Senate in 1992 to the term beginning January 1993; appointed by the Governor to begin serving on December 14, 1992, to fill the vacancy left by Senator Kent Conrad, who assumed the seat left vacant by the death of Quentin Northrop Burdick, and took the oath of office on December 15, 1992;

Pushed another commit: Forsythe died on 3/29/1984; the seat was vacant until November when Saxton finished Forscythe's term from 11/6 - 1/3, then started a full term of his own.
http://bioguide.congress.gov/scripts/biodisplay.pl?index=F000286


“Add this reply to my reply” failed to copy reference

Steps to reproduce

Login
Click "Reply to this message" in any article
Choose "My reply" in search box
Type in any keyword

After clicking “Add this reply to my reply” on searched result in editor, the contents of the copied reply appears in text area, but references shows "undefined".

Fixed by #358 . Kudos to @jojomango !


search fails

const collect = require('collect.js');
const collection = collect([2, 4, 6, 8]);
console.log(collection.search(item => item === 5));

return correctly false, but:
console.log(collection.search((item, key) => item === 5));
fails with: TypeError: Cannot read property 'toString' of undefined

Thanks for reporting.
Fixed in 9af320d41b75c836e9c9d23cc8850e41b19c85d0


Move CI to macOS 11.0 Big Sur

Running in parallel of #1276. I now wonder if the issue is not coming from having Xcode 12.2 with macOS 10.5. This moves up to macOS 11.0.

Not solving it either.


How to cancel the copy or cut operation?

What system are you running Yazi on?
Windows
What terminal are you running Yazi in?
Alacritty
Yazi version
0.1.5
Did you try the latest main branch to see if the problem has already been fixed?
Tried, but the problem is still present
Describe the bug
After copying or cutting the selected file, a yellow or red flag appears on the file. So how to cancel this copy or cut operation (i.e., remove the yellow or red flag in front of the file).
Just as the space bar selects a file, press the space bar again to deselect it. No shortcut to uncopy or uncut found. Also, there is no related process in the task manager. To cancel copy and cut, I can only exit yazi and enter again.
Expected Behavior
Whether the copy and paste shortcut keys can have the function of toggle.
To Reproduce
Open Yazi in terminal windows, select the file and enter the shortcut key for copying or cutting it.
Configuration
No response
Anything else?
No response

There's an RFC https://github.com/sxyazi/yazi/issues/319 that includes this feature request, and I will assess its feasibility after Yazi's plugin system is done.


Material-UI Enhanced Table: warnings in console

Describe the bug (required)
There are warnings in browser console.
Provide an example via Codesandbox! (required)
Use the react-table codesandbox to reproduce the issue.
Steps To Reproduce (required)
Steps to reproduce the behavior:

Go to https://react-table.js.org/examples/ui
Scroll down to Material-UI Enhanced Table
Click on Open in CodeSandbox
Click Console

Expected behavior (Recommended)
No warnings.
Screenshots

Desktop (please complete the following information):

OS: Windows 10
Browser: Vivaldi (chromium-based)
Version: 3.0.1874.23

This is an issue for MUI, not react-table. If it's a problem with the example, then I will accept a PR to update the example. Thanks!


Add classifier for Spring boot 2 jar

Failsafe Actuator dependency should have a classifier for Spring Boot 2 so that it stays backwards compatible.

Used classifier should be sb2.

See #74


name override.styl gives error in 1.0.0

So, changed the name to palette.styl and things work...

It currently depends on vuepress 0.x：https://vuepress.vuejs.org/default-theme-config/#simple-css-override


Create resource schema definition

Close #14
Overall explanation of your work :
Definition of the schema used to define workloads and resources.
Technical decisions you've made and why :
Tasks lists :

[ ] Create common resource schema
[ ] Create workload schema
[ ] Create user resource schema

The first proposal of Workload definition by @NilsPonsard  :
{
    "Workload" : {
        "name" : "",
        "type" : "",
        "executable" : "", // maybe change the name
        "ressources" : {
            "cpu" : "",
            "ram" : "",
            "disk" : "",
        },
        "port" : ""
    }
}

@NilsPonsard I think it would be better for us to discuss it here, upstream before we start committing, otherwise we may end up with a lot of commits to apply corrections.

I wanted to have a baseline to comment on and improve.

To match with scheduler/agent/controller .proto files, we should use something like this :
{
  "kind": "Workload", // This can be Workload, User, Network ...
  "type": "container",
  "name": "test-workload",
  "uri": "nginx",
  "ressources": {
    "cpu": 1,
    "ram": 2048,
    "disk": 2,
  },
  "ports": [
    "8080:80"
  ],
  "environment": [
    "NODE_ENV=production"
  ]
}

Do we need to add the replicas properties for Workload ?

What do you think? @do3-2021/kudo-controller-team

replicas property is not supported in v0 right ? So we'll add it later (when we'll add it to .proto & external API structures)

Do we all agree that the structure you proposed @iverly will be totally different if the kind is set to User ?

What I mean by that is all the structure depends on the type property
@Roxxas96 from what i understand the structure would depend on the kind property rather than the type property with @iverly ’s proposition.

What I mean by that is all the structure depends on the type property

@Roxxas96 from what i understand the structure would depend on the kind property rather than the type property with @iverly ’s proposition.

Yes my bad I meant the kind property

This PR is currently set to merge into main, is it okay or should we merge it in a team branch ?

You should merge it in a team branch. There is already a PR from the main branch to the parent's repo's main branch, merging now would add unrelated commits to the other PR.

This PR is currently set to merge into main, is it okay or should we merge it in a team branch ?

You should merge it in a team branch. There is already a PR from the main branch to the parent's repo's main branch, merging now would add unrelated commits to the other PR.

okay i changed it to the client team branch


ETA of 1.5 release?

want to use your library but having a lot of trouble since pre Meteor.startup() translations are not yet available.
I see there is a fix for this (issue #56) in v1.5  When do you think itll be released?

Hi,
The only reason it wasn't released already is a reported regression related
to use of i18next.addRule() in the application level, I am not sure when
I'll have time to give that issue the full attention it requires.
If you don't use the above it should be safe to use v1.5 (clone it to your
/packages dir).
-Daniel
On Wed, May 20, 2015 at 10:39 PM, Alex Corre<EMAIL_ADDRESS>wrote:

want to use your library but having a lot of trouble since pre
Meteor.startup() translations are not yet available.
I see there is a fix for this in v1.5. When do you think itll be released?
—
Reply to this email directly or view it on GitHub
https://github.com/TAPevents/tap-i18n/issues/75.

thanks @theosp should i use v1.5 or v1.5-rc2 branch?

rc2 .
On Thu, May 21, 2015 at 12:10 AM, Alex Corre<EMAIL_ADDRESS>wrote:

thanks @theosp https://github.com/theosp should i use v1.5 or v1.5-rc2
branch?
—
Reply to this email directly or view it on GitHub
https://github.com/TAPevents/tap-i18n/issues/75#issuecomment-104040497.


Fix pinning to treelite rather than lightgbm.

Currently the scheduled tests are failing:
improver_tests/calibration/rainforests_calibration/conftest.py:299: ModuleNotFoundError
=========================== short test summary info ============================
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test__check_num_features[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test__empty_config_warning[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test__evaluate_probabilities[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test__calculate_threshold_probabilities[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test__get_ensemble_distributions[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test_lead_time_without_matching_model[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test_process_ensemble_specifying_thresholds[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
ERROR improver_tests/calibration/rainforests_calibration/test_ApplyRainForestsCalibrationLightGBM.py::test_process_deterministic[treelite] - ModuleNotFoundError: No module named 'treelite_runtime'
= 5060 passed, 545 skipped, 1 xfailed, 5601 warnings, 8 errors in 196.43s (0:03:16) =
Error: Process completed with exit code 1.

The issue seems to be a change within treelite package. Fix to pin this package with #1966 pinned lightgbm rather than treelite. This PR is to unpin lightgbm and pin treelite.
Testing:

[x] Ran tests and they passed OK

Thanks @btrotta-bom.


sfsu: Add version 1.1.0

[X] I have read the Contributing Guide.

/verify

/verify Old hash files were in Powershell format not raw hashes

Just remove sfss.json? This seems not a deprecated.

The sfss and sfsl always give errors.

@niheaven do you have scoop installed in it's default location? they don't support non-default scoop locations yet.

Hmm, thanks, and non-default location should be taken into consideration IMO.

@niheaven Yeah I plan to add support in the near future, but in the meantime I will add better error message than that at least.

Just remove sfss.json? This seems not a deprecated.

Removed it altogether now


Mobile--- Out of Stock--- Looks funny with part picture and words

On the phones, it takes the images out and only leave a tiny bit at the top and it looks funny..
here is image from iPhone 11 Pro Max

here is image from Samsung 10+

When you look at the desktop it looks like this..... Can we get it to look like this..... - NOTE- we do NOT want the quick view just the " ! Temporarily out of stock message- Please check out other delicious products"

@monicakarnes
Bug Related in @165
Approach for Mobile to correct PLP search with OOS results
Response: I can create a style that hides the "Please check out our other delicious products" message on mobile on the PLP/Search/Product Cards only.  @monicakarnes confirm if this will work and noting we didn't have a mock-up for reference so may need to review with Sue/Mandi/Julie.  Your thoughts?

Enhancement Related in #165
we do NOT want the quick view just the " ! Temporarily out of stock message- Please check out other delicious products"
Response:  This enhancement isn't OOTB but if critical for Phase 1 we would need approval for ticket to investigate.  If not critical, suggesting to add to backlog for any changes once MVP in sprints are addressed

https://razoyo.atlassian.net/browse/TST-117

better for both, closing


Repeated labels on y-axes of cumulative charts on mobile

The y axis label is repeated when a user selects the cumulative chart for deaths within 28-days by date of death or by date of report on mobile devices.
The top two values both show 100k.
The y axis labels are incorrect on cumulative first dose vaccinations - 20 million and 30 million are both repeated. 15 million shows as 20 million and 25 million shows as 30 million.
Steps to reproduce the behaviour:

Go to deaths details page on mobile
Click on cumulative on the first chart
See error.

The same steps produce an error on the first dose vaccinations chart.
Expected behaviour
No repetition of y-axis labels.
Screenshots

Smartphone (please complete the following information):

Device: iPhone7
OS: iOS14.4.2
Browser: safari
Version: standard provided with iOS 14.4.2

Additional context
Could be a rounding error?

This also occurs on the vaccination uptake chart for second dose.

This one is incorrectly showing as 20M for values around 15M

This example shows 20M where it should be 15M


Cleanup toast loop

This unifies most of the toast logic into one retain call.
The motivation for this is related to #35.
Since dismisses have an animation, I check every frame whether the limit of notifications is reached and dismiss if so.
However if a toast has disappeared the old implementation called retain only in the next frame. Thus I was always overriding the disappeared state back to disappear.
As far as I can see from starting the example this is just as good. Why the dismiss (the variable) checked after the loop was not clear to me.

Thanks.


Created the useGet hook

This is a useGet hook that returns an array with three elements: response, loading, and error.

You can use the codesandbox compilation for testing if you like @LouieMartin

Linked with #12

Oh ok, I was also about to suggest to add unit testing but it looks like you have this


training Auxiliary classifier of Inception V3

Hello,
Since you are training the full network did you consider training the whole Inception-v3 along with the auxiliary classifier? I know Keras Inception V3 doesn't include the auxiliary classifier and also I understood from you implementation that you don't consider the auxiliary classifier.
Could that be a reason why replicating the JAMA paper is not possible so far? If not, did you manage to replicate the paper?
thank you and great work.

Hi, there was no mention of auxiliary classifiers used in the original paper, which is why we haven't considered using this. We assume the most significant reason why we could not reproduce JAMA's results is the data sets we used for training. They used two data sets from different institutions for training. We used a single data set for training obtained from Kaggle. They have also had a team of ophthalmologists to re-label their data sets, which may have significantly improved the quality of their data sets and therefore are getting better results.


Adding Login And SignUp Page

Hey I can Make Login And SignUp Page

Are you going to initilize backend for it?

I am interested in this issue please assign me @MastanSayyad

I am interested in this issue please assign me @MastanSayyad

thank you @kratik1237 but this issue is already assigned, kindly look for some other issues!

Okay @MastanSayyad

@kratik1237 if you can and Privacy Policy Page (which is empty in footer section) then raise the issue ill Assigned it to you!

Okay I do @MastanSayyad

Please assign me @MastanSayyad

Create a new issue for it, I'll assign it to you

Okay.
On Sat, Jun 29, 2024, 10:17 Mastan Sayyad @.***> wrote:

Create a new issue for it, I'll assign it to you @kratik1237
https://github.com/kratik1237
—
Reply to this email directly, view it on GitHub
https://github.com/MastanSayyad/Visual-Sort/issues/35#issuecomment-2197918017,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BB4QS3YN4LNB5WHYF3ILZWTZJY34LAVCNFSM6AAAAABJ4XMYSKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXHEYTQMBRG4
.
You are receiving this because you were mentioned.Message ID:
@.***>


authentication-policy.yaml mtls line "upstream errors" fix

This is a minor change to the the authentication-policy.yaml file.
https://github.com/apigee/istio-mixer-adapter/blob/master/samples/apigee/authentication-policy.yaml#L15
The following line
#  - mtls:   # uncomment if you're using mTLS between services in your mesh

should be changed to
#  - mtls: {}  # uncomment if you're using mTLS between services in your mesh

If you leave it as then you still get the "upstream connect error or disconnect/reset before headers" error.
- mtls:

When I execute kubectl get policy auth-spec -o yaml
partial output below.
peers:
  - {}

When I change the line to
- mtls: {}

and execute kubectl get policy auth-spec -o yaml then I get the following partial output:
peers:
  - mtls: {}

The documentation for enabling Istio mutual auth is shown below.
https://istio.io/docs/tasks/security/authn-policy/#globally-enabling-istio-mutual-tls

Thanks. Istio must have changed its interpretation of a nil config at some point. But using quotes is more correct, I'll change it.


Use broadcast for xsimd>=8

Checklist

[x] Added a CHANGELOG.rst entry

With bdf7fb40ad905c07fb557a67eb2bf518c48ce836, this should also work with jemalloc instead of jemalloc-local. I'm verifying this here.

Also, @xhochy: is it correct that you can't install tabmat in an environment where jemalloc is installed? Because then I get - package jemalloc-local-5.2.1-h9c3ff4c_6 has constraint jemalloc <0a0 conflicting with jemalloc-4.4.0-0

No, you can. You simply cannot combine jemalloc and jemalloc-local in an environment. For the runtime side, tabmat only requires libjemalloc-local which can live alongside jemalloc.


[research] Evaluate potential testing strategies for end-to-end testing Branch-Based Planner

Evaluate potential testing strategies for end-to-end communication between GitHub, informer, polling server, and the K8s API server.
End result sought: an outline/document covering how to test, what tools we have to use to mock GitHub to test end-to-end flow

To run an end-to-end test for the branch planner we need the following:

An existing Git repository which contains a Terraform CR and related Terraform definitions
A GitHub token with permissions to create pull requests and post comments to issues.

The test execution would look this:

Create a kind cluster
Bootstrap flux and point it to the above Git repository
Install tf-controller with branch planner enabled
Wait for Terraform CR condition to be set to Ready
Create a branch with a patch which updates the Terraform definition and push it to GitHub. Then create a PR from the branch.
Wait for a specified time period
Check that a comment has been posted to the PR

Create a branch with a patch which updates the Terraform definition and push it to GitHub. Then create a PR from the branch.

I wonder if we can just use one PR for everything and we don't have to create a new branch every time. The system checks configured repositories for PRs without corresponding TF object. On a new cluster it will be always the case.

@yitsushi It may be better to actually create a new PR and delete it after the test to keep things clean. Otherwise the test code needs to ensure that it checks the most recent comment from a long list of comments. Which is doable but makes troubleshooting the test a tiny bit more difficult.


Add adapter for Pest (PHP)

Simple change to the README to list an adapter for PestPHP.

Nice thanks for the PR! :smile:
Had a quick look and FYI instead of matching file path prefixes for a certain directory, you can define a filter_dir function so that only those directories are searched in the first place https://github.com/nvim-neotest/neotest/blob/master/lua/neotest/adapters/interface.lua#L17

Thanks for the merge and the excellent tip! I implemented it as you indicated, and it seems to speed things up, too!


No admin rights but Visual Studio already installed. Can I make it work?

I don't have admin rights, but I already have Visual Studio 2017 installed and usable in my computer.
However, node-gyp cannot find it for some reason.
Can windows-build-tools help me in this case? How?

same problem except that visual studio 2017 is not installed


MacOS Issue with python

Hello !
When running setup for ESP32 i get
✔ Here are the available target devices: · esp32
ℹ Ensuring esp32 install directory
⠸ Installing pyserial through pip3Command failed, Error: Command failed with exit code 1: python3 -m pip install pyserial

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: python3.13 -m pip install --upgrade pip
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:

    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with

    brew install pipx

    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.

    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.

    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
    at makeError (/Users/screamz/Library/pnpm/global/5/.pnpm/execa@5.1.1/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/Users/screamz/Library/pnpm/global/5/.pnpm/execa@5.1.1/node_modules/execa/index.js:118:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                               1.41s

Could be nice to use pipx instead ?
Also I have try to Create a virtual environment:
python3 -m venv myenv
source myenv/bin/activate
but later it crashes for other reason (related to virtual env ?)
✔ Here are the available target devices: · esp32
ℹ Ensuring esp32 install directory
✔ Installing pyserial through pip3
⠙ Installing esp-idf toolingDetecting the Python interpreter
Checking "python3" ...
⠹ Installing esp-idf toolingPython 3.13.0
"python3" has been detected
Checking Python compatibility
⠸ Installing esp-idf toolingInstalling ESP-IDF tools
Updating /Users/screamz/.espressif/idf-env.json
Selected targets are: esp32p4, esp32c5, esp32c3, esp32c6, esp32c61, esp32h2, esp32s3, esp32s2, esp32c2, esp32
Current system platform: macos-arm64
Installing tools: xtensa-esp-elf-gdb, riscv32-esp-elf-gdb, xtensa-esp-elf, riscv32-esp-elf, esp32ulp-elf, openocd-esp32, esp-rom-elfs
Installing<EMAIL_ADDRESS>Downloading https://github.com/espressif/binutils-gdb/releases/download/esp-gdb-v14.2_20240403/xtensa-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz
Destination: /Users/screamz/.espressif/dist/xtensa-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/xtensa-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz to /Users/screamz/.espressif/tools/xtensa-esp-elf-gdb/14.2_20240403
⠼ Installing esp-idf toolingInstalling<EMAIL_ADDRESS>Downloading https://github.com/espressif/binutils-gdb/releases/download/esp-gdb-v14.2_20240403/riscv32-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz
Destination: /Users/screamz/.espressif/dist/riscv32-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/riscv32-esp-elf-gdb-14.2_20240403-aarch64-apple-darwin21.1.tar.gz to /Users/screamz/.espressif/tools/riscv32-esp-elf-gdb/14.2_20240403
⠙ Installing esp-idf toolingInstalling<EMAIL_ADDRESS>Downloading https://github.com/espressif/crosstool-NG/releases/download/esp-13.2.0_20240530/xtensa-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz
Destination: /Users/screamz/.espressif/dist/xtensa-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/xtensa-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz to /Users/screamz/.espressif/tools/xtensa-esp-elf/esp-13.2.0_20240530
⠏ Installing esp-idf toolingInstalling<EMAIL_ADDRESS>Downloading https://github.com/espressif/crosstool-NG/releases/download/esp-13.2.0_20240530/riscv32-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz
Destination: /Users/screamz/.espressif/dist/riscv32-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz.tmp
Donestalling esp-idf tooling
⠴ Installing esp-idf toolingExtracting /Users/screamz/.espressif/dist/riscv32-esp-elf-13.2.0_20240530-aarch64-apple-darwin.tar.xz to /Users/screamz/.espressif/tools/riscv32-esp-elf/esp-13.2.0_20240530
⠸ Installing esp-idf toolingInstalling<EMAIL_ADDRESS>Downloading https://github.com/espressif/binutils-gdb/releases/download/esp32ulp-elf-2.38_20240113/esp32ulp-elf-2.38_20240113-macos-arm64.tar.gz
Destination: /Users/screamz/.espressif/dist/esp32ulp-elf-2.38_20240113-macos-arm64.tar.gz.tmp
Done Installing esp-idf tooling
Extracting /Users/screamz/.espressif/dist/esp32ulp-elf-2.38_20240113-macos-arm64.tar.gz to /Users/screamz/.espressif/tools/esp32ulp-elf/2.38_20240113
⠙ Installing esp-idf toolingInstalling<EMAIL_ADDRESS>Downloading https://github.com/espressif/openocd-esp32/releases/download/v0.12.0-esp32-20240318/openocd-esp32-macos-arm64-0.12.0-esp32-20240318.tar.gz
Destination: /Users/screamz/.espressif/dist/openocd-esp32-macos-arm64-0.12.0-esp32-20240318.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/openocd-esp32-macos-arm64-0.12.0-esp32-20240318.tar.gz to /Users/screamz/.espressif/tools/openocd-esp32/v0.12.0-esp32-20240318
⠧ Installing esp-idf toolingInstalling esp-rom-elfs@20240305
Downloading https://github.com/espressif/esp-rom-elfs/releases/download/20240305/esp-rom-elfs-20240305.tar.gz
Destination: /Users/screamz/.espressif/dist/esp-rom-elfs-20240305.tar.gz.tmp
Donestalling esp-idf tooling
Extracting /Users/screamz/.espressif/dist/esp-rom-elfs-20240305.tar.gz to /Users/screamz/.espressif/tools/esp-rom-elfs/20240305
⠇ Installing esp-idf toolingInstalling Python environment and packages
⠏ Installing esp-idf toolingCommand failed, Error: Command failed with exit code 1: ./install.sh
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
/Users/screamz/.local/share/esp32/esp-idf/tools/idf_tools.py:533: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  archive_obj.extractall(destination)
ERROR: This script was called from a virtual environment, can not create a virtual environment again
    at makeError (/Users/screamz/Library/pnpm/global/5/.pnpm/execa@5.1.1/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/Users/screamz/Library/pnpm/global/5/.pnpm/execa@5.1.1/node_modules/execa/index.js:118:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                              36.81s

I would be pleased to help for the macOS setup.
Pip seems managed by homebrew on my system.
xs-dev environment info:
  CLI Version                0.33.0
  OS                         Darwin
  Arch                       arm64
  Shell                      /bin/zsh
  NodeJS Version             v20.11.1 (/Users/screamz/.volta/tools/image/node/20.11.1/bin/node)
  Python Version              (/Library/Frameworks/Python.framework/Versions/2.7/bin/python)
  Moddable SDK Version       5.2.0 (/Users/screamz/dev-workspace/iot-moddable-workspace/moddable)
  Supported target devices   mac

If this is related to an error when using the CLI, please create an issue at "https://github.com/hipsterbrown/xs-dev/issues/new" with the above info.                                                                                918ms

[!NOTE]
Looks like there is a an issue with the CLI Code

https://github.com/HipsterBrown/xs-dev/blob/61ef0717ea02a3d7e776ad412438b5d78cf89b1b/src/toolbox/setup/esp32/mac.ts#L71-L76
This doesn't check for existence before install

Thanks for all this information @ScreamZ
It's interesting that you have python 2.7 on your system but not python 3 by default. This is a bad assumption in the CLI.

Could be nice to use pipx instead ?
Since pyserial is called by the underlying ESP-IDF tooling for flashing those devices, xs-dev is not able to swap out the pip install for pipx. But there should be additional checks before attempting that install step.

but later it crashes for other reason (related to virtual env ?)

Yes, the ESP-IDF install script will attempt to create a virtual environment for its dependencies and other scripts that get called when compiling and flashing a program.

I can confirm that switching to mise and installing python this way solved the issue.


CLAs signed status of the major contributors

The following table shows the CLAs signed status, including:

All the initial committers;
The major contributors who submit codes more than 300 lines. (The code line count statistics by GitHub)

Full Name
GitHub ID
ICLA signed
ICLA receipt
CCLA signed
CCLA receipt
email in the CLA file

肖宇
yu199195
✅
✅
✅
✅
<EMAIL_ADDRESS>

张永伦
tuohai666
✅
✅
✅
✅
<EMAIL_ADDRESS>

秦金卫
kimmking
✅
✅

<EMAIL_ADDRESS>

邓力铭
dengliming
✅
✅

<EMAIL_ADDRESS>

丁剑明
nuo-promise
✅
✅
✅
✅
<EMAIL_ADDRESS>

汤煜冬
tydhot
✅
✅

<EMAIL_ADDRESS>

冯振兵
fengzhenbing
✅
✅

<EMAIL_ADDRESS>

杨泽
Asxing
✅
✅

<EMAIL_ADDRESS>

张磊
SaberSola
✅
✅

<EMAIL_ADDRESS>

黄晓峰
huangxfchn
✅
✅

<EMAIL_ADDRESS>

王婷婷
fightingting
✅
✅

<EMAIL_ADDRESS>

刘伟
lw1243925457
✅
✅

<EMAIL_ADDRESS>

刘艺
Trafalgar-YuI
✅
✅

<EMAIL_ADDRESS>

刘伟
0x12FD16B
✅
✅

<EMAIL_ADDRESS>

宋明杰
KevinClair
✅
✅

<EMAIL_ADDRESS>

蒋晓峰
SteNicholas
✅
✅

<EMAIL_ADDRESS>

陈咸君
CCLooMi
✅
✅

<EMAIL_ADDRESS>

高猛
xlgm
✅
✅

<EMAIL_ADDRESS>

吴东东
wuudongdong
✅
✅

<EMAIL_ADDRESS>

吴雨宸
wyc192273
✅
✅

<EMAIL_ADDRESS>

江伟楷
MarcusJiang1306
✅
✅

<EMAIL_ADDRESS>

王真先
zendwang
✅
✅

<EMAIL_ADDRESS>

刘朋辉
Redick01
✅
✅

<EMAIL_ADDRESS>

汪鹏
BetterWp
✅
✅

<EMAIL_ADDRESS>

钟彦骏
Severezh
✅
✅

<EMAIL_ADDRESS>

伊文龙
yiwenlong
✅
✅

<EMAIL_ADDRESS>

胡文强
Wincher
✅
✅

<EMAIL_ADDRESS>

许晓东
xxd763795151
✅
✅

<EMAIL_ADDRESS>

梁自强
241600489
✅
✅

<EMAIL_ADDRESS>

刘良
midnight2104
✅
✅

<EMAIL_ADDRESS>

李菡
wuliendan
✅
✅

<EMAIL_ADDRESS>

feixue2008
could not be reached

薛永龙
xxl115
✅
✅

<EMAIL_ADDRESS>

王振武
cocoZwwang
✅
✅

<EMAIL_ADDRESS>

夏文涛
kminjava
✅
✅

<EMAIL_ADDRESS>

李明昊
lmhmhl
✅
✅

<EMAIL_ADDRESS>

cottom
could not be reached

乔帝鸽
strawberry-crisis
✅
✅

<EMAIL_ADDRESS>

骆潇龙
lxl910128
✅
✅

<EMAIL_ADDRESS>

刘耿华
kennhua
✅
✅

<EMAIL_ADDRESS>

钟金辉
onlyonezhongjinhui
✅
✅

<EMAIL_ADDRESS>

杜宇航
xiaoshen11
✅
✅

<EMAIL_ADDRESS>

范金鹏
hellboy0621
✅
✅

<EMAIL_ADDRESS>

林宽
Lin1nGithub
✅
✅

<EMAIL_ADDRESS>

帅仕
plutokaito
✅
✅

<EMAIL_ADDRESS>

周庭兆
Zhoutzzz
✅
✅

<EMAIL_ADDRESS>

那焱
naah69
✅
✅

<EMAIL_ADDRESS>

拱晓晨
magicalxiaochen
✅
✅

<EMAIL_ADDRESS>

周国栋
FocusZhouGD
✅
✅

<EMAIL_ADDRESS>

唐烨君
tyjwan
✅
✅

<EMAIL_ADDRESS>

宋跃全
songyuequan
✅
✅

<EMAIL_ADDRESS>

窦虎
hubao-1125
✅
✅

<EMAIL_ADDRESS>

唐甜
tangtian8
✅
✅

<EMAIL_ADDRESS>

唐震
ttttangzhen
✅
✅

<EMAIL_ADDRESS>

沈祥俊
stephenshen1993
✅
✅

<EMAIL_ADDRESS>

Liu Ying
marina432
✅
✅

<EMAIL_ADDRESS>

For the two contributors who could not be reached, we removed and refactored theirs codes.
All the commits of cottom https://github.com/dromara/shenyu/commits?author=cottom
The codes remove and refactor:
https://github.com/dromara/shenyu/commit/4aa398a
All the commits of feixue2008 https://github.com/dromara/shenyu/commits?author=feixue2008
The codes remove and refactor:
https://github.com/dromara/shenyu/commit/a972d63
https://github.com/dromara/shenyu/commit/4a624e5
https://github.com/dromara/shenyu/commit/c58b12d
https://github.com/dromara/shenyu/commit/4a70413
https://github.com/dromara/shenyu/commit/abcc02e

I just have a quick check with the icla signed information. I cannot find the these emails into the icla.txt.
Please check update the email with the one which sign the iCLA.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

hi, I'm sorry, the<EMAIL_ADDRESS>is error, and the right email for send ical is<EMAIL_ADDRESS>

hi, I'm sorry, the<EMAIL_ADDRESS>is error, and the right email for send ical is<EMAIL_ADDRESS>
Thx, I found iCLA record with<EMAIL_ADDRESS>

Hi @WillemJiang,
For<EMAIL_ADDRESS>and<EMAIL_ADDRESS>we can search the email address in<EMAIL_ADDRESS>where the receipt ack can be found. I think they should in the icla.txt if there are  acknowledgments.
Can you help to double check these emails? Maybe they are missed.

<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf. So use<EMAIL_ADDRESS>to search in the icla.txt may find the item<EMAIL_ADDRESS>is a valid mail address.

<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf<EMAIL_ADDRESS>is not a valid mail address because of typo. I suggested him to update the icla.pdf and re-send.

I am so sorry, the<EMAIL_ADDRESS>is error,  my correct email is<EMAIL_ADDRESS>

<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf. So use<EMAIL_ADDRESS>to search in the icla.txt may find the item<EMAIL_ADDRESS>is a valid mail address.

yes, 2 email addresses are valid, I used<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf

I just have a quick check with the icla signed information. I cannot find the these emails into the icla.txt.
Please check update the email with the one which sign the iCLA.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
I am so sorry, I sent a email which attachment appears to be unsigned. I have sent another email, but there is no reply.

I am so sorry, the<EMAIL_ADDRESS>is error, my correct email is<EMAIL_ADDRESS>
Please resend the iCLA with right email address with it.

<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf. So use<EMAIL_ADDRESS>to search in the icla.txt may find the item<EMAIL_ADDRESS>is a valid mail address.

yes, 2 email addresses are valid, I used<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf

Please resend the iCLA with right email address with it.

I just have a quick check with the icla signed information. I cannot find the these emails into the icla.txt.
Please check update the email with the one which sign the iCLA.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
I am so sorry, I sent a email which attachment appears to be unsigned. I have sent another email, but there is no reply.

Could you send the iCLA again?

OK，I was sent the correct icla pdf
发自我的iPhone
------------------ Original ------------------
From: Willem Jiang @.>
Date: Mon,May 31,2021 8:59 PM
To: dromara/shenyu @.>
Cc: liupenghui @.>, Comment @.>
Subject: Re: [dromara/shenyu] CLAs signed status of the major contributors (#1461)
I am so sorry, the @.*** is error, my correct email is @.***
Please resend the iCLA with right email address with it.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf. So use<EMAIL_ADDRESS>to search in the icla.txt may find the item<EMAIL_ADDRESS>is a valid mail address.

yes, 2 email addresses are valid, I used<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf

I can find the<EMAIL_ADDRESS>in the icla.txt file, if you are using this mailbox, we just need to update the table file for it.

Thanks for the help of WillemJiang and the ASF secretary,  all of the problems have been solved:
For 1 & 2, Willem confirmed these two by alternative emails.
For 3 & 4, The ASF secretary received the iclas and sent the receipt acknowledgement.

[ ] 1<EMAIL_ADDRESS>is another email of Liang Liu. While the email signed in icla is<EMAIL_ADDRESS>.
[ ] 2<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf.
[ ] 3<EMAIL_ADDRESS>sent the icla file, but<EMAIL_ADDRESS>is written in the icla.pdf<EMAIL_ADDRESS>is not a valid mail address because of a typo. I suggested Penghui Liu update the icla.pdf and re-send.
[ ] 4<EMAIL_ADDRESS>Dongdong Wu signed his name typed in a script font that does not constitute a signature. I suggested that he update the icla.pdf and re-send.

I just checked the iclas.txt file, this below emails are in the iclas.txt
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

@tuohai666  Please update those emails into the table.

@WillemJiang Done. I have just update those emails into the table.


Automatic type conversion for whole number floats

Type conversions are a very convenient feature in Robot framework. To make them that extra bit convenient I propose an enhancement.
Currently passing any string representation of a float number to a keyword accepting only int will fail. In most cases this is justified, but there are situations where floats are convertible to int just fine. Examples are "1.0", "2.00" or "1e100". Note that these conversions currently are already accepted when passed as type float (i.e. ${1.0} or ${1e100}.  Conversion for numbers for which the decimal part is non-zero should still fail. We are talking about conversion here, not type casting.

Yes, this makes sense. From user perspective it's odd we accept the ${2.0}but not2.0`. This is so easy to change we can still do it in RF 7.0. As a bonus this helps mitigating problems caused by changes to internal conversion APIs that some libraries have used.


test(intervals): add test cases for fromSemitones

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

5 of 5 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 88.636%

Totals

Change from base Build<PHONE_NUMBER>:
0.04%

Covered Lines:
273

Relevant Lines:
308

💛  - Coveralls


Added Token Validations

Feature or Bugfix

Bugfix

Detail

Add GET to /oauth/userInfo/ endpoint of IdP using accessToken to determine token validity

Relates

Security
Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

Does this PR introduce or modify any input fields or queries - this includes
fetching data from storage outside the application (e.g. a database, an S3 bucket)?

Is the input sanitized?
What precautions are you taking before deserializing the data you consume?
Is injection prevented by parametrizing queries?
Have you ensured no eval or similar functions are used?

Does this PR introduce any functionality or component that requires authorization?

How have you ensured it respects the existing AuthN/AuthZ mechanisms?
Are you logging failed auth attempts?

Are you using or adding any cryptographic features?

Do you use a standard proven implementations?
Are the used keys controlled by the customer? Where are they stored?

Are you introducing any new policies/roles/users?

Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Testing Completed:

[ ] CodePipeline Deploys Successfully (for Cognito Auth with already existing data.all instance - i.e. UPDATE)

[ ] Login + Store Access Token

[ ] SignOut (clear tokens)

[ ] ReAuth Workflow Successfuly

[ ] Post SignOut Run data.all API with same Tokens -- returns _______

[ ] User removed from Cognito IdP --> User no longer can perform actions on data.all UI immediately


[TableCard] add dynamic link columns

Internal Watson-IoT/pal-tracking#390

Summary
Users would like to be able to make table columns into dynamic links. For example, in monitor, the user may want to link to their own website / dashboards with specific, unique information such as the assetId to get further information about the device.
Additional context
Internal issue: https://github.ibm.com/wiotp/monitoring-dashboard/issues/330

:tada: This issue has been resolved in version 2.74.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


docs: update handbook troubleshooting docs to include sherif as mon…

Update docs
Update handbook troubleshooting docs to include sherif as monorepo linter.
Description
I started using https://github.com/QuiiBz/sherif in our projects. Then I stumbled across the troubleshooting page in the turbo docs (https://turbo.build/repo/docs/handbook/troubleshooting). It already mentions @manypkg/cli as monorepo linter. This PR adds sherif docs as a fast, rust based alternative.

Hey, thanks for this! We have a major rewrite currently in progress for the documentation so we're not adding any non-essential new content to the current documentation.
I'll revisit this PR once we're done there and get back with you!

We did an entire docs rewrite that now includes sherif! Thanks for pushing for the mention.


[cluster-test] New knob --threads-per-ac for smaller validator sets

This will allow to customize number of threads per AC.

Will need this knob for pre-mainnet test, as we going to have fewer validators for now
Also might help with perf test on small validator count

By default the value is automatically calculated to make a best guess of optimal number of threads per AC:

We don't want too many threads, so we limit it to 10 per AC
We want at least 1 thread per AC
We want to make sure that on average we have about 75-150 AC threads total
We want to have equal number of threads for each AC

This LGTM.
Can you also add a way to control EmitThreadParams, right now they are hardcoded for experiments and there is no way to change them.

We can do it in separate diff :) I don't want to overload this one

@bors-libra r=andll

:pushpin: Commit e7b8d54 has been approved by andll

I keep messing up bors commands
@bors-libra r+

:bulb: This pull request was already approved, no need to approve it again.

There's another pull request that is currently being tested, blocking this pull request: #2440

:pushpin: Commit e7b8d54 has been approved by ankushagarwal

:hourglass: Testing commit e7b8d547e185e5c92bae0fd53796a54e2db5be90 with merge 2a77bd1305dae469c4dfd2f1f084f438a7f84326...

:sunny: Test successful - checks-circle_commit_workflow
Approved by: ankushagarwal
Pushing 2a77bd1305dae469c4dfd2f1f084f438a7f84326 to master...


ocrConfidence is always 0.0

I am getting ocrConfidence 0.0 everytime from below method
ScanResults results = data.getParcelableExtra(IntentUtils.DATA_EXTRA);
results.ocrConfidence();

We are looking into this issue. Thank you

This has been addressed in the latest version of the SDK.


"Too many open files (os error 24)" when using --threads 4 or greater (or 0)

Using the 2013 JSONL file from https://data.open-contracting.org/en/publication/63
flatterer --threads 0 --force -j paraguay_dncp_records_2013.jsonl out

This is an operating system level error as there is a limit to operating system open file descriptors.  It is working on my 8core/16 thread machine with --threads 0.
You can raise the limits, there are ones at the user level and the system level:
http://woshub.com/too-many-open-files-error-linux/
I do not think I can lower the files that are open in this mode with flatterer.  It will open (threads*amount of tables) file descriptors so it could add up to quite a bit.  However, it is likely that it is some other processes are also using a lot and flatterer is just pushing it over the limit as it opens up a lot quickly.
It may be possible for flatterer, to not hold on to the file descriptors and reopen the file each time, as an option, but I would have to look at performance implications of that.

Hmm, on macOS, running:
launchctl limit maxfiles

yields
maxfiles    256            unlimited

256 is a soft limit, unlimited is a hard limit. I guess Rust respects the soft limit? There are 65 files that are ultimately being written, so I guess that explains why it errors with 4 threads.

Setting ulimit -n 1024 fixes the issue.

I think a note would be good. I don't know if that error can be caught in Rust – if so, a note can be made along with the exception.

I have captured the exact error code and gave a clearer error message for that case:
https://github.com/kindly/libflatterer/commit/04cfa739a2155fa7f1c8c7d4ae78f5f90421f904
This will be in the next release.


[d3d12-memory-allocator] Update to 2.0.1

Updates to latest tagged release. Now uses the provided cmake file instead of using a custom one (although a custom target file was added)

[*] Changes comply with the maintainer guide.
[*] SHA512s are updated for each updated download.
[*] The "supports" clause reflects platforms that may be fixed by this new version.
[*] Any fixed CI baseline entries are removed from that file.
[*] Any patches that are no longer applied are deleted from the port's directory.
[*] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[*] Only one version is added to each modified port's versions file.

The usage test passed on x64-windows (header files found):
d3d12-memory-allocator provides CMake targets:

  # this is heuristically generated, and may not be correct
  find_package(d3d12-memory-allocator CONFIG REQUIRED)
  target_link_libraries(main PRIVATE unofficial::D3D12MemoryAllocator)


Calculate and populate Paypal fees

Docker is using Paypal to receive money. Currently, we only calculate and populate fees automatically for Stripe transactions.
For collectives receiving money from Paypal, we need to:

calculate host fees, platform fees, and txn fees
Investigate how paypal handles transactions in different currency (meaning, currency other than the currency of the account). what information does it give us?
Populate backend transaction table with fees and currency info

@asood123 this we'll need to do...

We are not allowing paypal donations any more.


[question] Unicode and non-unicode support?

Can someone please clarify how jwt-cpp supports Unicode or non-unicode (ANSI) implementations, in terms of:
(I am specifically referring to using this on Windows).

Which builds (Unicode or Non-Unicode) of OpenSSL are required to use jwt-cpp, or will it work with either?
Will jwt-cpp compile equally well in Unicode vs non-Unicode environments? I notice no specific dependencies on the UNICODE defines and minimal reference to wstring/wchar etc.
Obviously the JWT itself is encoded as Base64Url, but once the token is decoded into a JSON object, are the individual traits represented only as UTF-8? Or is this a function of the JSON library used (we'll probably stick with picojson initially).

Thanks

Hi,
jwt-cpp doesn't support Unicode, because JWT's are defined to use UTF-8. UTF-8 is generally useful for these kinds of tasks because it is equivalent to ASCII for most of the letters which are usually found in a token and has the nice property of not containing 0 bytes (meaning ascii functions like strlen work as expected). Most json libraries should be binary safe (meaning if your string contains 0 bytes they should be preserved) but they return a std string. Basically you have two options:

Provide a json_trait with a string type that implicitly converts between utf8 and unicode.
Use utf8 for the jwt and convert to/from it on your api, c++ provides wstring_convert for this.

I'd recommend going with the second approach because it's easier to maintain and probably more performant.

The original reason I added the traits was for cpprestsdk, which on Windows uses a wstring... There are still parts of the API the are locked to std::string and I know of other implementations which went with approach two.
If you are looking to contribute, wide string support would be amazing ❤


T-SNE visualization

I want to visualized data after a training network. could you provide an idea of how can I visualize source and target data after training for comparison of data alignment way? I read your paper you are using T-SNE for visualization but I have no idea how can I use T-SNE after training finished. I prepared T-SNE before training source and target data but after training, I have a problem with visualization.

I use tensorboard-X.
You just need to feed data and domain labels to the embedding API of tensorboard, then you can generate the t-SNE visualization.


Research/fix geolocation

https://github.com/jmathai/elodie/issues/203
RETHREAD
Hey all,
Is anyone experiencing difficulties generating (geo)location file names?
Sample file(s):

Hope to hear back
Cheers

It's working for me. What's your output with debug=True in constants.py?
./elodie.py import --destination=/Users/jaisen/Downloads/test /Users/jaisen/Downloads/source/baf32efe-1521-11e7-8218-54cb8fd2546d.jpeg --allow-duplicates
/Users/jaisen/Downloads/source/baf32efe-1521-11e7-8218-54cb8fd2546d.jpeg -> /Users/jaisen/Downloads/test/2017-03-Mar/Mġarr/2017-03-11_11-35-37-baf32efe-1521-11e7-8218-54cb8fd2546d.jpeg

No debug messages by the looks of it?
Cheers

You've set debug=True and your mapquest key is correct in your config.ini file? Can you confirm if your location.json file has an "Unknown Location" value?

(Will revoke that key later)
Cheers

Fixed it
config.ini was in the wrong place
https://github.com/jmathai/elodie#using-openstreetmap-data-from-mapquest - I shan't be so complacent next time and read the instructions every time I install
Hope this is useful to someone else
Apologies

👍 I think config.ini belongs in the root of the repository. The *.json files probably do as well.


feat!: Upgrade module to include capacity providers, cloudwatch log group, and minimum supported versions

Description

Minimum supported version of Terraform AWS provider updated to v4.6 to support the latest resources utilized
Minimum supported version of Terraform updated to v1.0
ecs-instance-profile sub-module has been removed; this functionality is available through the terraform-aws-modules/terraform-aws-autoscaling module starting with version v6.5.0. Please see the examples/ec2 example for a demonstration on how to use and integrate with the terraform-aws-autoscaling module.
The container_insights and capacity_providers variables have been replaced by new variables

See UPGRADE-4.0.md for more details
Motivation and Context

Update module to use current GA versions of provider and Terraform
Incorporate functionality to support EC2 AutoScaling Group capacity provider
Provide complete functionality for ECS cluster

Closes #56
Closes #58
Closes #59

Move IAM instance profile out to AutoScaling group module where it is commonly used (by ECS, EKS, or standalone)
Lays the groundwork for adding service & task definition functionality in sub-module

Breaking Changes

Yes, see UPGRADE-4.0.md for more details

How Has This Been Tested?

[x] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[x] I have tested and validated these changes using one or more of the provided examples/* projects
[x] I have executed pre-commit run -a on my pull request

Need to make a few small adjustments to ASG module to better support a cleaner upgrade path for instance profiles