Refactor icon build process

Description
This PR refactors the way icons are built and persisted.  While our current strategy "works" most of the time, it continues to have periodic issues (see #1080, #1254, #1336, #1626, #2149, #2199.  This PR both dramatically simplifies the icon colorizing & compiling process (it includes #2279, which should be reviewed/merged first), and (I hope) will reduce errors as follows:
The vast majority of the errors we've seen are related to locating the pyrcc5 or pyside2-rcc executable scripts in the environment (currently done with _find_rcc_or_raise).  This PR avoids calling those python scripts as subprocesses and just uses the direct mechanism used by the corresponding Qt package:

for PyQt5, pyrcc5 really just directly calls PyQt5.pyrcc_main.processResourceFile (which directly calls a shared library function rather than executing a binary)
for PySide2, pyside2-rcc in turn just calls the non-python resource compiler binary (rcc -g python) that comes packaged at the top level of the PySide2 directory.

By calling these directly, we should be able bypass the need for a successful installation of a "console_script" in the python environment. (Basically, if PyQt5 or PySide2 can be imported at all, these should work... which can't be said for our current strategy).
This PR also builds towards allowing anyone (including plugin developers) to call register_qt_resources such that they too can use custom SVGs (matching the theme) in qss style sheets.
I also refined the hashing mechanism recreating resources.  It now only rebuilds if the contents of the icons directory has changed, or if a new version of PyQt5/PySide2 is being used.
In a later PR, I can also add a step to add resource compilation to our setup.py packaging so that it's included in the distribution.
Type of change

[ ] Bug-fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

References

How has this been tested?

[ ] example: the test suite for my feature covers cases x, y, and z
[ ] example: all tests pass with my change

Final checklist:

[ ] My PR is the minimum possible work for the desired functionality
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works

OK, I updated this a bit to allow anyone (plugins) to be able to register icons and use them in stylesheets as follows:
from napari.qt import compile_qt_resources

compile_qt_resources(svg_paths=['my_icon.svg'], colors=['#3956C9'], register=True)

see working example in examples/custom_icon.py.
This is all set for review. (it looks like a lot of lines to review), but it's mostly just docstrings.  This cuts actual lines of code by more than half.

thanks @sofroniewn I added a relatively thorough module docstring explaining _register_napari_resources for internal users, and mentioning compile_qt_svgs for external users, and made _temporary_qrc_file private, and added an __all__ attribute for clarity

Ok, these changes look good to me. I'll wait to see if anyone else has additional comments, and if not will merge tomorrow

@tlambert03 now that we added the examples/dev folder do you want to more the custom_icon.py and grin.svg in there too? You don't have too as they use napari, but I do see them as more developer oriented examples than user oriented examples. Let me know what you think? I'm fine either way

yep, I like it

merged master and moved the example to /dev

Great stuff here. I will merge now. Follow up PR can be made to address the doc string/ return type mismatch

errors about pyside-rcc might definitely arise when building icons.  Can you open a new issue, give us information on your platform and how you created the environment (and how you installed your qt backend), and what the full error was?


ubpm: 1.7.3 -> 1.10.0

Description of changes
https://codeberg.org/LazyT/ubpm/releases
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

I see darwin build failing and it was failing at https://github.com/NixOS/nixpkgs/pull/224593
should mark as broken on darwin ?

Probably yes, as I'm not sure how to fix it and don't have mac. So marked.


A way to avoid calling ignite cache too often for quick queries

should also not bother calling ignite for data when total rowcount is 0

@chrisjstevo had one idea which was to use something similar to LogAtFrequency (currently in the code base to log at certain intervals)
It is a good option but i dont think we can call something similar in the IgniteOrderDataProvider because one instance of this is shared across multiple instances of the VirtualizedSessionTable


[Feature] Extend the podman API to create/delete/start/stop/list LLM servers (= containers)

Is your feature request related to a problem? Please describe
As a developer/coder developing applications using AI (langchain4j, etc) it is not so simple locally to play with LLM (server, model, etc). Even if podman desktop allows to list/install models like to create/delete servers, the users should use the UI for that purpose and cannot use an API
The purpose of this feature is to extend the podman API to create/delete/start/stop/list LLM servers (= containers)
Describe the solution you'd like
n/a
Describe alternatives you've considered
n/a
Additional context
n/a

Planning this in 1.6 milestone

cc @maxandersen

Cool - any plan of enabling #1787 ? Main diff is that this issue just talks about creating/stopping ai services that is on different ports - #1787 is about proxying so paths and ports are stable.


[SPARK-26315][PYSPARk] auto cast threshold from Integer to Float in approxSimilarityJoin of BucketedRandomProjectionLSHModel

What changes were proposed in this pull request?
If the input parameter 'threshold' to the function approxSimilarityJoin is not a float, we would get an exception.  The fix is to convert the 'threshold' into a float before calling the java implementation method.
How was this patch tested?
Added a new test case.  Without this fix, the test will throw an exception as reported in the JIRA. With the fix, the test passes.
Please review http://spark.apache.org/contributing.html before opening a pull request.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #4469 has started for PR 23313 at commit eb00074.

Test build #4469 has finished for PR 23313 at commit eb00074.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@srowen Thanks for committing this PR!


Bluetooth: BAP: Dont discover ASE CP if ASE not found

Added a check in unicast_client_ase_discover_cb that if no ASE was discovered, then it would stop the discovery there instead of attempting to call
unicast_client_ase_cp_discover to discover the control point which would not be useful to use anyhow.
This terminates the discovery earlier in case of the remote side not supporting the audio direction we are discovering.

Rebased to solve merge conflict


[SPARK-29037][Core] Spark gives duplicate result when an application was killed

What changes were proposed in this pull request?
Case:
Application appA insert overwrite table table_a with static partition overwrite.
But it was killed when committing tasks,  because one task is hang.
And parts of its committed tasks output is kept under /path/table_a/_temporary/0/.
Then we run application appB insert overwrite table table_a with dynamic partition overwrite.
It executes successfully.
But it also commit the data under /path/table_a/_temporary/0/ to destination dir.
In this PR, we skip the FileOutputCommitter.{setupJob, commitJob, abortJob} operations when dynamicPartitionOverwrite.
Why are the changes needed?
Data may corrupt without this PR.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing Unit Test.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #110617 has started for PR 25795 at commit c3aff4e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15593/
Test PASSed.

Test build #110617 has finished for PR 25795 at commit c3aff4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110617/
Test PASSed.

retest this please

@dongjoon-hyun
Thanks for your suggestion.
I have moved the UT to sql/core module.
And I found that the committer here  is an instance of org.apache.parquet.hadoop.ParquetOutputCommitter .
And ParquetOutputCommitter is only used in sql/core module.
So, I think we only need add UT to sql/core module.
Can you help take a look again? Thanks in advance!

Can one of the admins verify this patch?

Test build #110637 has started for PR 25795 at commit 804ac8c.

Test build #110637 has finished for PR 25795 at commit 804ac8c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110637/
Test FAILed.

The failed test org.apache.spark.SparkContextSuite passed locally.
retest please.

Test build #110725 has started for PR 25795 at commit 144ec63.

Test build #110725 has finished for PR 25795 at commit 144ec63.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110725/
Test PASSed.

cc @vanzin @steveloughran

Hi, @cloud-fan and @gatorsmile .
Could you review this PR, please?

Test build #110760 has started for PR 25795 at commit 979fbe4.

The standard "commit by rename" FileOutputCommitters assume exclusive access to the output directory tree; if they find existing attempt data, it's assumed to be from a previous attempt of the same query, or from a different query. MapReduce Will try to use that previous attempt data, otherwise ignore it. At the end of the query everything under _temporary Will probably be deleted. That's an implicit concurrency filter. Spark has historically complicated life within attempt ID of 0.
I'm not actually sure how the different commit algorithms would cope with concurrent output. It's implicit in the V2 algorithm that different tasks can commit simultaneously, but since that algorithm lacks the failure semantics spark expects (task commits are not atomic) I'm not going to recommend it. And the v1 algorithm probably assumes it has exclusive access... You'd have to review its job commit algorithm in more detail to be confident that things will not go horribly wrong (see https://github.com/steveloughran/zero-rename-committer/releases/download/tag_draft_005/a_zero_rename_committer.pdf)
The S3A committers probably could support concurrent access if every Application attempt had a unique ID, and they opted not to attempt cleanup by deleting the attempt directories and all in progress multipart uploads under a path. You'd better make sure that your bucket had an object life cycle to delete outstanding uploads, otherwise you get an extra large bill of the end of the month. But people should be doing that anyway.
What we do have there is Ryan's Partioned Committer which is designed to perform in situ updates of an existing table by only worrying about conflicts in the final partition of the directory tree. You can choose a policy of what to do if that is partition exists: fail, append or replace: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/PartitionedStagingCommitter.java#L122
Sparked dynamic partition code seems to depend on rename being fast, but as it's done on a per file basis, doesn't rely on atomic directory commit (I think; I've not looked at it in enough detail). Which means, it's not that good for S3 at all. And as the output committer doesn't know the final destination of the output, there's no obvious way to move to an object-store specific commit mechanism. I just steered clear of it. Now could more be done spark-side if filesystems actually made their store specific upload operations visible?  But I am going to be ruthless here: is effort Best spent on trying to tweak how we write to object stores, or on object-store-first Data structures such as Iceberg et al.

Test build #110760 has finished for PR 25795 at commit 979fbe4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110760/
Test FAILed.

thanks @steveloughran for the detailed explanation!
Seems it's very complicated to make Spark support concurrent and object-store friendly files writing. We should ask users to try object-store-first Data structures such as Delta and Iceberg.
I'd like to narrow down the scope. Spark should at least guarantee data correctness. Duplicated data should not happen. @turboFei @advancedxy is it possible to detect concurrent writes and fail the query?

I think it is feasible to detect concurrent writes.
As known, when dynamicPartitionOverwrite is true, the job attempt path is .spark-staging-${UUID}.
when dynamicPartitionOverwrite is false, it is  _temporary.
We just need to check the existences of these paths and throw exception to fast fail when concurrent writes are detected.

when dynamicPartitionOverwrite=true, we still set up the output committer, will that create the _temporary dir?

BTW .spark-staging-${UUID} is not a concern. UUID should never conflict.

when dynamicPartitionOverwrite=true, we still set up the output committer, will that create the _temporary dir?

No, setupCommitter will not create the _temporary dir, it is created by committer.setupJob(jobContext).

BTW .spark-staging-${UUID} is not a concern. UUID should never conflict.
Yes, UUID is not a concern, but we can check the directory name whether started with .spark-staging.

We just need check the existences of these paths and throw exception to fast fail when concurrent writes are detected, Include:

dynamicPartitionOverwrite and dynamicPartitionOverwrite
dynamicPartitionOverwrite and staticPartitionOverwrite/non-partitioned table write
staticPartitionOverwrite/non-partitioned table write and staticPartitionOverwrite/non-partitioned table write

Yeah, this is what we discussed offline. To fail fast,  Spark should check output existence to detect concurrent writes to the same partitions.

Seems it's very complicated to make Spark support concurrent and object-store friendly files writing. We should ask users to try object-store-first Data structures such as Delta and Iceberg.

Yes, it's hard for Spark to support concurrent writes. But the scope of https://github.com/apache/spark/pull/25739 and this one is to support concurrent writes to different partitions, which can be a nice feature and not that hard to achieve.

@advancedxy can you give a completed proposal for it? Basically the requirement:

concurrent writing to the same non-partitioned table should fail
concurrent writing to different partitions should success
concurrent writing to the same partition should fail

Test build #110799 has started for PR 25795 at commit f0b6c84.

Test build #110799 has finished for PR 25795 at commit f0b6c84.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110799/
Test FAILed.

@advancedxy can you give a completed proposal for it?

All right, I think the requirements can be split into two parts:

support concurrent writes to different locations(partitions).
It's achieved by setting different output path for different writes:

For dynamicPartitionOverwrite, the output could be the staging dir(current solution of #25739), which is unique from each other.
For  dynamicPartitionsOverwrite=false and partitioned table, the output in the OutputCommitter could be $table_output/static_part_key1=value1/static_part_key2=value2/.... Concurrent writes to partitions prefixed by different static partitions won't interfere each other. This could be extended in #25379.
For non-partitioned table, there's only one output, don't support concurrent writes.

detect concurrent writes to the same location and fail fast.
This can be archived during setupJob stage. We can check the existence of output path like the FileOutputFormat did. If the output path has already been existed, it must be created by other concurrent writing job or left by previous failed/killed job. We can throw an exception with the possible reasons and fails the current job. Of course, we cannot simple check the output passed to JobConf as the $table_output should be presented(unless the first time to create table). $table_output/_temporary/$app_attempt_num could be a good candidate.
One more thing to do in Spark, spark should infer yarn app attempt num when running under yarn mode. Currently, the app attempt num is always 0 when writing.

I believe the approach proposal should covers concurrent writes and case in this pr. WDYT @cloud-fan, @turboFei and @wangyum

SGTM

So #25739 is to support concurrent writes to different locations, and this PR is to detect concurrent writes to the same location and fail fast?

So #25739 is to support concurrent writes to different locations, and this PR is to detect concurrent writes to the same location and fail fast?

I am fine with that as long as @turboFei is ok.

So #25739 is to support concurrent writes to different locations, and this PR is to detect concurrent writes to the same location and fail fast?

So #25739 is to support concurrent writes to different locations, and this PR is to detect concurrent writes to the same location and fail fast?

I am fine with that as long as @turboFei is ok.

Firstly, for dynamicPartitionOverwrite, it should skip FileOutputCommitter. {setupJob/commitJob/abortJob}. And the concurrent support should be resolved by https://github.com/apache/spark/pull/25739 @advancedxy .
Then when dynamicPartitionOverwrite=false , if I detect there is sub dir under $table_output/_temporary exists, it should fast fail.

Then when dynamicPartitionOverwrite=false , if I detect there is sub dir under $table_output/_temporary exists, it should fast fail.

With my knowledge.
When enableDynamicOverwrite is false.
Case:
tbl_a(c1 int, c2 int) partitioned by (c2)
For the sql: insert overwrite table tbl_a select ...
I can not detect it's destination partition.

I have discussed with advancedxy offline, and I am clearly for the solution now.
Thanks @advancedxy .  I will complete this PR after https://github.com/apache/spark/pull/25739 is merged.
Thanks again.

We can check the existence of output path like the FileOutputFormat did.

How does it work for dynamicPartitionOverwrite=true? The output is staging dir which is unique. Can we detect 2 jobs that are writing to the same table with dynamicPartitionOverwrite=true?

How does it work for dynamicPartitionOverwrite=true? The output path is staging dir which is unique. Can we detect 2 jobs that are writing to the same table with dynamicPartitionOverwrite=true?

Looks like there is no easy way to detect concurrent writes with dynamicPartitionOverwrite=true.
Maybe we should notice user about that limitation.
User won't get duplicated result in this case, but the result could be messed(some part replaced by other part, while some part remains) when writing concurrently since no transaction is involved.

@cloud-fan wrote

Seems it's very complicated to make Spark support concurrent and object-store friendly files writing.

Mostly it's about S3, that being the hard one to work with. GCS is consistent; file rename O(1); dir rename atomic O(files). Azure Datalake Gen2 (ABFS connector) works exactly like a real filesystem. But they all have latency issues, directory listing overheads, and the other little costs that make switching to them tricky.
S3 is really a fault injection framework for all your assumptions about what a store is.

We should ask users to try object-store-first Data structures such as Delta and Iceberg.

+1
What those object-store-first Data structures do promise is not just correctness in the presence of that fault injection layer, they scale well in world where enumerating files under a directory tree is no longer a low-to-medium cost operation.
w.r.t dynamic partition updates, for AWS S3 the best thing to do may be to say "don't"; at least not this way. But you still have to get it right everywhere else.

Spark should at least guarantee data correctness.

I concur.
Now, one thing stepping through the FileOutputFormat code in a debugger while taking notes showed me is that the critical commit algorithms everyone relies on are barely documented and under-tested when it comes to all their failure paths. And once you realise that the v2 commit algorithm doesn't meet the expectations of MR or Spark, you worry about the correctness of any application which has used it.
Now seems a good time, to not just get the dynamic partition overwrite code to work -but to specify that algorithm enough to be able to convince others of its correctness on a consistent file system. I did try to start with a TLA+ spec of what S3 did, but after Lamport told me I misunderstood the nature of "eventually" I gave up. I hear nice things about Isabelle/HOL these days -maybe someone sufficiently motivated could try to use it.
BTW, if you haven't read it, look at the Stocator: A High Performance Object Store Connector for Spark by @gilv et al, who use task ID information in the file names to know what to roll back in the presence of failure. That is: rather than trying to get commit right, they get abort right.
(returns to debugging a console log where, somehow, cached 404s in the S3 load balancer are still causing problems despite HADOOP-16490 fixing it).

For the dynamicPartitionOverwrite=true, I propose my thought  here:
We can unify the staging dir for both dynamic and static insert overwrite, base on the specifed  static partition keyValue pairs.
  private def stagingDir = {
    val stagingPath = ".spark-staging-" + jobId + "/" +
    staticPartitionKVS.map(kv => "sp_" + kv._1 + "=" +
kv._2).mkString(File.separator)
    new Path(path, stagingPath)
}

For example below:
ta (c1 int, p1 int, p2 int, p3 int) partitioned by (p1, p2, p3)
insert overwrite table ta partition(p1=1,p2,p3) select ...
// stagingDir: .spark-staging-${UUID}/sp_p1=1

insert overwrite table ta partition(p1=1,p2=2,p3) select ...
// stagingDir: .spark-staging-${UUID}/sp_p1=1/sp_p2=2

insert overwrite table ta select ...
// stagingDir: .spark-staging-${UUID}

The stagingDir will be cleaned up after job finished.
Before per insert, we should check the path whose name is started with .spark-staging and find the longest path with sp_ prefix.
For two paths A and B.
If A is fully contained by  B or B is fully contained by A,  they can not concurrent write.
For example:
A: 'sp_p1=1/sp_p2=2'  B: 'sp_p1=1'   A is fully contained by B.
A: ''  B: 'sp_p1=1'   A contains all other paths.
A: 'sp_p1=1/sp_p2=2'  B: 'sp_p1=1/sp_p2=3'   A and B do not contain each other.
About  whether use a UUID in staging dir, It  is determined  by the implementation of https://github.com/apache/spark/pull/25739 .
In addition, this PR is dedicated to resolve the duplicat result issue originally.
I have created a new PR https://github.com/apache/spark/pull/25863 for duplicate result issue, hope it can make sense. Thanks.

Before per insert, we should check the path whose name is started with .spark-staging and find the longest path with sp_ prefix.

The problem here is that, how do you detect insert overwrite table ta select ...? .spark-staging-${UUID} always exists if there is an ongoing write.

The problem here is that, how do you detect insert overwrite table ta select ...? .spark-staging-${UUID} always exists if there is an ongoing write.

We will find a longest  path with sp_ prefix.
For ``insert overwrite table ta select ..., its output path is .spark-staging-${UUID}and its longest  path withsp_prefix is""`.
Then we can detect that there is a write operation for whole table, all other write operation for this table are not allowed.

Is there any way to block on the commit process to ensure that exactly one can be committing at the same time, eg. some lease file with a timestamp inside whose (unexpired) presence is a sign someone else is committing? for all filesystems where create(path, overwrite=false) is atomic you can be confident when you create the file that you have the permission to commit. For others, you either need to fail, spin, or decide when it is so out of date that you could delete it and go for the lock yourself.
(I'm ignoring inconsistent storage without CRUD consistency or atomic creates, obviously)

Thanks @steveloughran for mentioning our paper on Stocator.  (This is a better reference -- Stocator: Providing High Performance and Fault Tolerance for Apache Spark Over Object Storage.)
It is better to characterize Stocator wrt commit rather than abort.   Stocator only commits.  The last commit wins and any parts that do not belong are ignored when the partition is read (and they can also be deleted at that time to reclaim space or space can be reclaimed by an independent garbage collection process).  This can be achieved by writing a manifest at the time of commit that has a list of the parts that belong to the commit, e.g., by extending the _SUCCESS file/object.  And then when the partition is read, the parts listed in the manifest of the most recent commit are read.  Maybe a similar solution would be easier here, rather than trying to ensure only one commit at a time.

@hkkolodner -thanks for the clarification.
FWIW, in https://github.com/apache/hadoop/pull/1442 I am actually stripping back on enumerating all files in the _SUCCESS manifest I've been creating (for test validation only) because at a sufficiently large terasort and TCP-DS scale, you end up with memory issues in job commits, which makes my colleagues trying to do these things unhappy. I may be overreacting (it's happening at an earlier stage), but I'm just being thorough in reviewing datastructures built in a commit. And even if there's enough heap, I don't want to force a many-MB upload at the end of every query, as that becomes a bottleneck of its own.
Like I say, the post-directory-listing table layouts are the inevitable future. That directory tree has been great: tool neutral, easy to navigate by hand, etc, but it hits scale limits even in HDFS, serious perf limits in higher-latency stores and the commit-by-rename mechanism is running out of steam.
I don't have enough experience of any of the alternatives to have any strong opinions, except to conclude that yes, they are inevitable.


Coin Lockup Scaffolding in ConnectTransaction, DB Utils, and UTXO View

Current dependencies on/for this PR:

feature/coin-lockups

PR #678

PR #679

PR #680   👈

This comment was auto-generated by Graphite.


Restrict labels to remain within a desired region/view.

This is a very useful controller. It would be even more useful if it supported gesture restriction for a specified view.

Hi @thedp,
Can you explain in more detail? I'm sorry, but I don't quite understand what you mean by "gesture restriction".

I'll try to explain.
Let's say I have a UIImageView, and I want the labels to be placed only within the region of that specific view. Currently, the controller allows the dragging of the label anywhere on the screen.

Ok,  I thought so.

i hope too

@thedp I have added property enableMoveRestriction in last version and update demo


papermc: 1.17.1r97 -> 1.17.1r399

Motivation for this change
Fixes #149983
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Would someone mind testing this again? I left home this evening and don't have access to a machine to run this on at the moment

Result of nixpkgs-review pr 149989 run on x86_64-linux 1

1 package built:

papermc

Server works as expected.

Should we backport this because it fixes a RCE security issue?


Fix grid roles for accessbility

Related to #1199 which was said to be fixed by #1208 but the PR only addressed the issue with Tables and not Grids.
This PR fixes a default role and adds another role to each cell which allows screen readers to correctly read Grids. Without this change, screen readers don't recognize any cells within the Grid and fails to announce them (tested with Voice Over on Firefox and Chrome).

[x] The existing test suites (npm test) all pass
[x] For any new features or bug fixes, both positive and negative test cases have been added
[ ] For any new features, documentation has been added
[ ] For any documentation changes, the text has been proofread and is clear to both experienced users and beginners.
[x] Format your code with prettier (yarn run prettier).
[x] Run the Flow typechecks (yarn run typecheck).

Last publish was Dec 2022... can we get a release?


The error message from PackageLoader could be more helpful

When following the basics of setting up jinja2, if you don't create the appropriate templates directory in your package, you get this error message:
ValueError: The 'mypackage' package was not installed in a way that PackageLoader understands.

This could be significantly improved by explaining the actual problem: The PackageLoader could not find a suitable "templates" directory.
To replicate, add this to mypackage.py:
from jinja2 import Environment, PackageLoader

env = Environment(loader=PackageLoader("mypackage"))

If you run python mypackage.py you will see a traceback similar to this:
Traceback (most recent call last):
  File "mypackage/mypackage.py", line 4, in <module>
    env = Environment(loader=PackageLoader("mypackage"))
  File ".virtualenvs/mypackage/lib/python3.10/site-packages/jinja2/loaders.py", line 291, in __init__
    import_module(package_name)
  File ".pyenv/versions/3.10.3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "mypackage/mypackage.py", line 4, in <module>
    env = Environment(loader=PackageLoader("mypackage"))
  File ".virtualenvs/mypackage/lib/python3.10/site-packages/jinja2/loaders.py", line 323, in __init__
    raise ValueError(
ValueError: The 'mypackage' package was not installed in a way that PackageLoader understands.

Seeing an error message that explained that jinja2 couldn't find an appropriate "templates" directory would be much better.
Environment:

Python version: 3.10.3
Jinja version: 3.1.2

Happy to review a PR.


Cannot enable plugins on aliases

Describe the bug
Plugins cannot be enabled from the edit alias page.
To Reproduce
Steps to reproduce the behavior:

Ensure a plugin enabled and configured from the settings page.
Ensure you are in a new session and have not tried to create a new alias first.
Edit an existing Alias
Scroll down to Plugins and attempt to click Enable sending messages that match this alias via <plugin name>
The tick box will not tick and an error is produced in the browser debug console

Expected behavior
The tickbox should tick and the options should appear for that plugin.
Screenshots

Environment information:

OS:  Official Docker image running on Ubuntu x86 (same issue in non official image)
DB: Maria DB running in Docker.
Accessed via  Nginx reverse proxy over HTTPS with custom host name.
Other details: Confirmed to occur in Firefox and Chromium based browsers

Additional context
Works as expected if you first try to add an alias and tick the box. You don't have to save that alias, just use the feature. After that if you try to edit an existing alias it will work as expected for a while. refreshing the browser appears to allow the error to return.
Error from the browser console:
TypeError: Cannot create property 'Pushover' on string '"\"\\\"{\\\\\\\"Discord\\\\\\\":{},\\\\\\\"Filter\\\\\\\":{},\\\\\\\"Gotify\\\\\\\":{},\\\\\\\"MessageRepeat\\\\\\\":{},\\\\\\\"MicrosoftTeams\\\\\\\":{},\\\\\\\"Pushover\\\\\\\":{},\\\\\\\"RegexReplace\\\\\\\":{},\\\\\\\"SMTP\\\\\\\":{},\\\\\\\"Shell\\\\\\\":{},\\\\\\\"Slack\\\\\\\":{},\\\\\\\"Telegram\\\\\\\":{},\\\\\\\"Template\\\\\\\":{},\\\\\\\"Twitter\\\\\\\":{}}\\\"\""'
    at fn.assign (eval at compile (angular.js:15358:15), <anonymous>:4:1047)
    at Object.$$writeModelToScope (angular.js:28810:5)
    at angular.js:28804:9
    at g (angular.js:28723:9)
    at e (angular.js:28706:9)
    at Object.$$runValidators (angular.js:28650:5)
    at Object.$$parseAndValidate (angular.js:28791:5)
    at Object.$commitViewValue (angular.js:28757:5)
    at angular.js:28898:9
    at b.$eval (angular.js:18161:16)

This is likely a result of dirty data. Deleting the alias and creating it again fixes the issue. Not worth fixing this.


Start-UnityEditor fails if there are compiler errors even when passing -ErrorAction 'SilentlyContinue'

With previous versions of Unity, passing 'SilentlyContinue' to -ErrorAction would cause Unity to ignore initial compiler errors. This is impotant in cases where the Unity is expected to run code which will retrieve compile time dependencies. In Unity 2020, 'SilentlyContinue' doesn't seem to affect Unity and Start-UnityEditor will fail.
I experienced this issue using Nuget4Unity, which is a plugin that performs a nuget package restore after Unity starts. Since Unity fails to start now, the packages are never restored and Start-UnityEditor cannot be used.
Repro Steps:
-Create an empty Unity 2020 project (I am using Unity 2020.3.20f1)
-Import the Nuget4Unity plugin
-Add a Nuget package
-Create a script which references a .net type from the Nuget package
-run "sue [YourProjectPath] -AcceptAPIUpdate -BatchMode -Quit -Wait -BuildTarget 'WSAPlayer' -ErrorAction SilentlyContinue
Expected Result:
UnitySetup emits some errors about references to the types found in the Nuget package and continues to build the project.
Actual Result:
UnitySetup emits some errors about references to the types found in the Nuget package and then exits.
I reproduced this bug in both the release version 5.4.140 and the pre-release version 5.5.146-develop.

A user can circumvent this issue by using the -AdditionalArguements paramter with "-ignoreCompilerErrors" at the bottom of this page.
https://docs.unity3d.com/2020.2/Documentation/Manual/SafeMode.html

Very strange - SilentlyContinue is a PowerShell construct and not a UnitySetup feature. As long as we're using Write-Error for the errors then PowerShell should continue running and not log the errors.


Change log level dynamically not working

I'm trying out the basic example, specifically the functionality to change the log level dynamically as described here: https://github.com/SoftwareAG/cumulocity-microservice-templates/tree/main/basic#changing-the-log-level-dynamically
However, this does not seem to work. Steps I've did:

Clone repository on main branch
do mvn clean install
create the application in my tenant & subscribe
get bootstrap credentials and add them to application-dev.properties
making sure there's management.endpoints.web.exposure.include=health, loggers line in application-dev.properties
Run build: java -Dspring.profiles.active=dev -jar cumulocity-microservice-templates-basic-0.0.1-SNAPSHOT.jar

Service runs fine on my localhost, a curl http://localhost:8080/health returns up. However, none of these calls returning logger info:

curl http://localhost:8080/loggers
curl http://localhost:8080/loggers/com.c8y.ms.templates.basic.service
curl http://localhost:8080/actuator/loggers

Am I missing something here? Is it working on your side?

Found the problem and it was quite obvious: I was missing authentication.
curl http://localhost:8080/loggers -u '{tenant}/{user}:{pass}' works fine.


Update to Python 3.7

Bit of a strawman PR to muse over upgrading to Python 3.7. AFAIK there shouldn't be (m)any incompatibilities that we need to update, but I'm going to play around with it locally and see if any problems arise. Interested to know if we think this is worth doing at all.
Summary
Bump us up onto Python 3.7. This brings a set of new features/guarantees
described here: https://www.python.org/downloads/release/python-370/
Notable changes include:

Addition of a debugger() shortcut.
Order-preserving dict insertions are now guaranteed.
Data classes
Avoiding ascii encoding by default; now utf-8.

Note: In the process of recycling my venv, I noticed that I didn't seem to be able to reinstall deps from scratch and that our pinned version of pyminiracer seemed incompatible with pip10/python37 (not sure which/what). Be interested to know, also, if anyone else has this problem?

@samuelhwilliams This affects me a bit less, but I'm up for moving to the latest Python release. UTF8 encoding seems like the biggest win.
I've not had any issues installing pyminiracer recently, but then I've not reinstalled my virtualenv for a while.
Should we look at switching to pipenv for managing dependencies too? Maybe that would help to ensure that we're all using the same version of all the things?

I finally got round to trying this... couldn't get our pinned psycopg2 version to install with Python 3.7 but update to latest version is all good.
Once I got that sorted the py-mini-racer also needed to be updated as in this PR.
And then I needed to install a bunch of python header files to get wheels to build, but I think that is probably just Ubuntu.

Sorry wrong button :man_facepalming:


Checkstyle: accept cr or lf as last character in generated files

Currently, checkstyle has an issue with generated code that yields a warning that the file does not end with CRLF (but apparently with CR or LF). The checkstyle code can be made a bit more lenient.

Change in dsol-checks.xml:
  <module name="NewlineAtEndOfFile">
    <property name="severity" value="warning"/>
    <property name="lineSeparator" value="lf_cr_crlf"/>
    <property name="fileExtensions" value="java"/>
  </module>

By the way, to allow all three is also the default checkstyle setting at the moment.


refactor(common): move comm protocols and motor control to common

Overview
Rather than copying over firmware files into each subproject, we should put common firmware files in the common subproject so that other subprojects can simply add those files to their individual executables.
Next Steps

We should add message types for the gantry subsystem

Review Requests
Does the structure seem OK? Any change requests or files to be moved around?

A few questions from the noob.

How do we decide if something needs to be in the common project? Just if more than one other subproject uses it?
It looks like can_task.cpp is a straight copy and paste of can_task.cpp in /gantry/firmware. Can we pull this file out into common to be used by multiple pieces of firmware?

A few questions from the noob.

How do we decide if something needs to be in the common project? Just if more than one other subproject uses it?
It looks like can_task.cpp is a straight copy and paste of can_task.cpp in /gantry/firmware. Can we pull this file out into common to be used by multiple pieces of firmware?

Yep, if more than one subproject uses a file and it can be separated out then it should go in common. Things that cannot be separated out typically rely heavily on parts of the BSP for either FreeRTOS or an ST board.
can_task.cpp falls into the category that it cannot be separated out due to it relying directly on FreeRTOS.

Is the idea that the common/firmware files are the default configurations for the various peripherals? These can be overriden by not compiling them into the sub-project?

Is the idea that the common/firmware files are the default configurations for the various peripherals? These can be overriden by not compiling them into the sub-project?

Yep that's correct


Merge pull request #16 from CristianTafur249/predev

Predev to develop to production

It's not necessary


Add country, streaming-bytestring

Made a https://github.com/michaelt/streaming-bytestring/pull/20

streaming-<IP_ADDRESS> (Kostiantyn Rybnikov<EMAIL_ADDRESS>@k-bx) is out of bounds for:

[ ] streaming-bytestring-<IP_ADDRESS> (>=<IP_ADDRESS> && < <IP_ADDRESS>). Kostiantyn Rybnikov<EMAIL_ADDRESS>@k-bx. @michaelt. Used by: library, test-suite

All right, should be fixed now!

Oops, updated the haskell-src-exts constraints, somebody please restart travis to try again

It passed!

Merged, thanks
(BTW you can trigger  travis by reopening the PR.)

(BTW you can trigger travis by reopening the PR.)

Great to know, thanks!

Building test suite 'test' for friday-juicypixels-<IP_ADDRESS>..

test/Spec.hs:1:1: error:
    File name does not match module name:
    Saw: ‘Main’
    Expected: ‘Spec’
  |
1 | {-# LANGUAGE TemplateHaskell #-}


Declaring a function inside a try/catch prohibited by "no-inner-declaration"

I find myself wanting to create a function inside a try/catch block. For example:
async function uploadImages () {
  const profileUrl = buildProfileUrl()

  const picturesUrl = buildPicturesUrl()

  try {
    const profile = await getProfile(profileUrl)

    const pictures = await getPictures(picturesUrl)

    async function upload (picture) {
      return uploadImage({ profile, picture })
    }

    const promises = pictures.map(upload)

    return Promise.all(promises)
  } catch (error) {
    throw error
  }
}

I want to declare upload inside the try so I can take advantage of profile being in scope. However, this apparently violates the no-inner-declarations rule.
Can someone explain why this rule is important here? Should no-inner-declaration perhaps not apply to try/catch?

You can fix this by making your upload function into an expression instead.
const upload = async function (picture) {
  return uploadImage({ profile, picture })
}

However, this raises a good point. Now that ES6 has shipped everywhere, block-scoped functions are supported widely. Perhaps we can disable this rule now? Anyone have thoughts?

It seems that indeed functions should be allowed. From the rule docs:

The function declaration portion rule will be rendered obsolete when block-scoped functions land in ES6, but until then, it should be left on to enforce valid constructions.

Regarding variables, perhaps we should allow them as well and so disable this rule and enable block-scoped-var, instead?

Disable checking variable declarations when using block-scoped-var or if declaring variables in nested blocks is acceptable despite hoisting.

Regarding variables, perhaps we should allow them as well

Variables are allowed. This rule is set to only prevents function declarations.

re: block-scoped-var

I'd like to move to const/let only so that eliminated the need for such a rule. #633

Thank you for the clarifications, @feross.
Then, it seems to be clear that we just drop this rule and enable no-var, right?

Yep.

This will ship in standard v15.


chore(@ngtools/webpack): allow JSON files to be cacheable

Reason
Static JSON files put in the assets folder are not currently cacheable because they are ignore by the Webpack url-loader.

Files and directories in the .angular-cli.json assets array are not meant to be processed through loaders.
I'm not too clear on what's the usecase you're trying to fix here, and what broken. Can you elaborate please? Opening an issue would be better than discussing here as well.

Sure. See this related issue https://github.com/angular/angular-cli/issues/7505