Added supports Driver method to DatabaseConnection

Environment
Liquibase Version: 4.3.2
Liquibase Integration & Version: maven
Liquibase Extension(s) & Version: liquibase-mongodb-4.3.2
Database Vendor & Version: MongoDB 4.0.3
Operating System Type & Version: Windows
Pull Request Type

[x] Bug fix (non-breaking change which fixes an issue.)
[ ] Enhancement/New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description
This is the initial fix in core. It is required a fix for the extensions as well.
First commit introduces a support(Driver) method in DatabaseConnection interface with default implementation to return true.
This is in order to not be a breaking change. Otherwise it is required to be implemented in any Implementations.
Subsequent commits has to fix Executor injection based on support(Database) implementation. This part has to be proved as not working after the initial commit.
Exception for the wrong Executor after Connection fixed:
	at liquibase.nosql.executor.NoSqlExecutor.queryForObject(NoSqlExecutor.java:80) 	at liquibase.database.AbstractJdbcDatabase.getConnectionSchemaName(AbstractJdbcDatabase.java:364) 	at liquibase.database.AbstractJdbcDatabase.getDefaultSchemaName(AbstractJdbcDatabase.java:325) 	at liquibase.database.AbstractJdbcDatabase.escapeObjectName(AbstractJdbcDatabase.java:919) 	at liquibase.snapshot.SnapshotGeneratorFactory.has(SnapshotGeneratorFactory.java:114) 	at liquibase.snapshot.SnapshotGeneratorFactory.hasDatabaseChangeLogLockTable(SnapshotGeneratorFactory.java:279)
Fixes #1782
Fixes #1783
Steps To Reproduce
-run update for a postgres or other db having liquibase-mongodb.jar in classpath
Actual Behavior
Exception described in the issues linked
Expected/Desired Behavior
Successful run
┆Issue is synchronized with this Jira Story by Unito

After fixing the Connection an additional commit should be about injecting the right Executor by Database implementation.
Currently with this fix only it throws:
	at liquibase.nosql.executor.NoSqlExecutor.queryForObject(NoSqlExecutor.java:80) 	at liquibase.database.AbstractJdbcDatabase.getConnectionSchemaName(AbstractJdbcDatabase.java:364) 	at liquibase.database.AbstractJdbcDatabase.getDefaultSchemaName(AbstractJdbcDatabase.java:325) 	at liquibase.database.AbstractJdbcDatabase.escapeObjectName(AbstractJdbcDatabase.java:919) 	at liquibase.snapshot.SnapshotGeneratorFactory.has(SnapshotGeneratorFactory.java:114) 	at liquibase.snapshot.SnapshotGeneratorFactory.hasDatabaseChangeLogLockTable(SnapshotGeneratorFactory.java:279)

It this falls into a larger refactoring we are planning around the Database/DatabaseConnection APIs. There is a bunch of changes we need to think about to better support nosql databases etc. and this PR would go along with that larger effort.
We’re already working through a large refactoring in the liqubiase.command and liquibase.configuration APIs for the in-progress 4.4.0, but am hoping to get this PR and the other related ones in with 4.5.0.

Hi @alexandru-slobodcicov
Thanks for providing this fix for these two issues.
As Nathan stated, we are looking to get this in fairly soon once we get 4.4.0 out the door.
Here's what happens next.
A member of the Liquibase team will take a look at your contribution and may suggest:

Changes based on guidelines for PR submission
Additional tests (unit and/or integration) and/or Test Harness Tests
Provide other feedback, like requesting changes.

The PR will be prioritized according to our internal development and testing capacity.

I changed the supports() method from taking a driver to taking a URL. As we get into NoSQL databases, we are tryign to get away from the java.sql  package as an assumption, so don't want to add more dependencies on it.


support items[] properties

Hi，items[] properties seems not to be supported.
the JsonSchema4 file looks like this below
{
"type": "object",
"properties": {
"fixedItemsList": {
"type": "array",
"title": "A list of fixed items",
"items": [
{
"title": "A string value",
"type": "string",
"default": "lorem ipsum"
},
{
"title": "a boolean value",
"type": "boolean"
}
],
"additionalItems": {
"title": "Additional item",
"type": "number"
}
}
}
}
you can also find it in tab[Arrays] from https://mozilla-services.github.io/react-jsonschema-form/
It can't be opened in VisualJsonEditor.
Regards.

Same issue to nestedList.
{
"type": "object",
"properties": {
"nestedList": {
"type": "array",
"title": "Nested list",
"items": {
"type": "array",
"title": "Inner list",
"items": {
"type": "string",
"default": "lorem ipsum"
}
}
}
}
}

Tuple types (items with different type per element) are currently not supported.

What about nested arrays?
Array-object is ok, but array-array is not, because the inner has no properties.

Probably not, currently only the simplest scenarios are supported..


Use CAPABILITY_NAMED_IAM for manual deployment

Without it, the deployment fails when I accept all the default settings from the SAM CLI.
This was already mentioned in issue #64.
See below for complete dialog with sam deploy --guided that ends in failure.
I used SAM CLI, version 1.89.0 and repo version 415ea5a255c08a2b4858cfb081d35c8706451c2f.

Configuring SAM deploy
======================

	Looking for config file [samconfig.toml] :  Not found

	Setting default arguments for 'sam deploy'
	=========================================
	Stack Name [sam-app]: assignmentgroup-macro
	AWS Region [us-east-1]:
	Parameter NumChildStacks [-1]:
	Parameter MaxAssignmentsAllocation [-1]:
	Parameter LookupNames [false]:
	Parameter DefaultSessionDuration []:
	Parameter ChildTemplatesInYaml [false]:
	Parameter MaxConcurrentAssignments [-1]:
	Parameter MaxResourcesPerTemplate [-1]:
	Parameter LogLevel [INFO]:
	Parameter ArtifactS3KeyPrefix []:
	Parameter S3PutObjectArgs []:
	#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
	Confirm changes before deploy [y/N]: y
	#SAM needs permission to be able to create roles to connect to the resources in your template
	Allow SAM CLI IAM role creation [Y/n]: Y
	#Preserves the state of previously provisioned resources when an operation fails
	Disable rollback [y/N]:
	Save arguments to configuration file [Y/n]: Y
	SAM configuration file [samconfig.toml]:
	SAM configuration environment [default]:

	Looking for resources needed for deployment:
	Creating the required resources...
	Successfully created!

	Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-b231npc8gr37
	A different default S3 bucket can be set in samconfig.toml and auto resolution of buckets turned off by setting resolve_s3=False

	Saved arguments to config file
	Running 'sam deploy' for future deployments will use the parameters saved above.
	The above parameters can be changed by modifying samconfig.toml
	Learn more about samconfig.toml syntax at
	https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

	Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  10060041 / 12681481  (79.33%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  10322185 / 12681481  (81.40%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  10584329 / 12681481  (83.46%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  10846473 / 12681481  (85.53%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  11108617 / 12681481  (87.60%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  11370761 / 12681481  (89.66%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  11632905 / 12681481  (91.73%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  11895049 / 12681481  (93.80%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  12157193 / 12681481  (95.87%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  12419337 / 12681481  (97.93%)       Uploading to assignmentgroup-macro/034b0fe011f7287db8c89ec9a8c1f4d9  12681481 / 12681481  (100.00%)

	Deploying with following values
	===============================
	Stack name                   : assignmentgroup-macro
	Region                       : us-east-1
	Confirm changeset            : True
	Disable rollback             : False
	Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-b231npc8gr37
	Capabilities                 : ["CAPABILITY_IAM"]
	Parameter overrides          : {"NumChildStacks": "-1", "MaxAssignmentsAllocation": "-1", "LookupNames": "false", "DefaultSessionDuration": "", "ChildTemplatesInYaml": "false", "MaxConcurrentAssignments": "-1", "MaxResourcesPerTemplate": "-1", "LogLevel": "INFO", "ArtifactS3KeyPrefix": "", "S3PutObjectArgs": ""}
	Signing Profiles             : {}

Initiating deployment
=====================

	Uploading to assignmentgroup-macro/bbb778716c67f5899bd34816c4b31cb3.template  5537 / 5537  (100.00%)

Waiting for changeset to be created..

Error: Failed to create changeset for the stack: assignmentgroup-macro, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state: For expression "Status" we matched expected path: "FAILED" Status: FAILED. Reason: Requires capabilities : [CAPABILITY_NAMED_IAM]

Ah, thanks for this.

No, thank you! Your macro is already supporting the SSO infrastructure on my latest project. As I'm learning how it all works this documentation helps me to get familiar with it.


Add-on is not compatible with Vaadin 24.4

Describe the bug
After upgrading the pom to 24.4.0.beta5, starting the demo fails with the following message:
(index):6 Uncaught (in promise) DOMException: Failed to execute 'define' on 'CustomElementRegistry': the name "vaadin-button" has already been used with this registry
    at window.customElements.define (http://<IP_ADDRESS>:8080/:6:829)
    at VAADIN/build/generated-flow-imports-DRaju_tl.js (http://<IP_ADDRESS>:8080/VAADIN/build/generated-flow-imports-DRaju_tl.js:3522:20)
    at __require (http://<IP_ADDRESS>:8080/VAADIN/build/generated-flow-imports-DRaju_tl.js:3:50)
    at http://<IP_ADDRESS>:8080/VAADIN/build/generated-flow-imports-DRaju_tl.js:44377:16

Expected behavior
The demo should start with no issues
Minimal reproducible example

Set vaadin.version in v24 profile to 24.4.0.beta5 and add the following dependency

<dependencies>
    <dependency>
	<groupId>com.flowingcode.vaadin.addons.demo</groupId>
        <artifactId>commons-demo</artifactId>
        <version>4.0.0</version>
    </dependency>
</dependencies>

Run mvn jetty:run -Pv24

Add-on Version
1.0.2-SNAPSHOT
Vaadin Version
24.4.0.beta5
Additional information
No response

Was able to reproduce it just by adding the add-on dependency in a simple Vaadin 24.4.0.beta5 project. Not sure why it is happening though. It could be the fact that the web-component has vaadin components pinned in version 23.1.5? Or that it is using lit version 2? I tried several beta versions and it keeps failing. In 24.3.12 works fine.

https://github.com/FlowingCode/fc-whatsapp-button/blob/dca3024c8416b44307ff482dee2e46ed1c5e98d7/package.json#L35-L39 has pinned dependencies "23.1.5".
See https://github.com/FlowingCode/FontAwesomeIronIconset/issues/105 for a related issue

I updated vaadin dependencies to version ^23.3.35 in the web-component part and did a new release https://github.com/FlowingCode/fc-whatsapp-button/releases/tag/1.0.4. But when updating the server part to use this new version of the web-component, problem still persists.

As a follow up to my previous message, the problem is that updating the web-component part to ^23.3.35 is not going to work for vaadin 24.4 as ^ only includes versions in the 23 range. So it will try to use version 23.5.2:
npm why node_modules/@flowingcode/fc-whatsapp-button/node_modules/@vaadin/button
<EMAIL_ADDRESS>node_modules/@flowingcode/fc-whatsapp-button/node_modules/@vaadin/button
<EMAIL_ADDRESS>from<EMAIL_ADDRESS>  node_modules/@flowingcode/fc-whatsapp-button
<EMAIL_ADDRESS>from the root project

We need to create a new version of the web-component compatible with 24.4 and then update the flow part.


Change the place where the packages_dir is created.

For #2 , I fixed to create packages_dir before calling the function plugin.cpt_build().

fixed in a34c4b08d8c564058a2449c77996504d994ef551


[zh] sync 2023-02-06-k8s-gcr-io-freeze-announcement.md

Fix #39381
Add zh text to this blog:
content/zh-cn/blog/_posts/2023-02-06-k8s-gcr-io-freeze-announcement.md

See preview.

/lgtm


Add a dedupe integration

Problem Statement
Some of Sentry SDKs have a dedupe integration which prevents the SDK from sending multiple instances of the same exception.
JS: https://docs.sentry.io/platforms/javascript/configuration/integrations/dedupe/
Python: https://docs.sentry.io/platforms/python/integrations/default-integrations/#deduplication
Solution Brainstorm
Create the same integration for flutter to not send duplicated events.
Are you willing to submit a PR?
None

This is already implemented in the Dart SDK: https://github.com/getsentry/sentry-dart/blob/main/dart/lib/src/event_processor/deduplication_event_processor.dart

@rodolfoBee did the flutter dedup not work for you?


Add Jira Dashboard plugin

Hey, I just made a Pull Request!
This PR adds the Jira Dashboard plugin. The plugin is a Jira issue overview that can be added as a page in the EntityPage overview tabs. The issue view displays issue summaries to help achieve better task visibility and more efficient project management.

Information that this plugin provides:

The Jira project that is relevant for that entity, with information about the project name, key, category, description, project lead, and avatar.
Jira issues and their type, key, summary, status, and assignee.
By default, the issue views that are provided are incoming issues, open issues and assigned to you. You can also customise the view to display the information relevant for your entity by defining Jira filters and components.

Isn’t there already a Jira plugin?
Yes, Roadie is one of the main Backstage contributors and offers the Backstage Jira plugin. This plugin also provides information from Jira, such as project and project type.
However, Roadie’s Backstage Jira plugin focuses on exposing the quantity of different issue types and the project activity stream. The new Jira Dashboard plugin in this PR takes a different approach, displaying issues in tables. In addition, the Jira Dashboard plugin provides the possibility of linking not only a Jira project, but also specific Jira components and filters to your entities.
Changes:

Backend: Added the Jira Dashboard backend plugin
Frontend: Added the Jira Dashboard frontend plugin

:heavy_check_mark: Checklist

[X] A changeset describing the change and affected packages. (more info)
[X] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[X] Screenshots attached (for UI changes)
[X] All your commits have a Signed-off-by line in the message. (more info)

Hi @fridajac, I know you have comments in the PR description about this but have you actually reached out to Roadie to see if they would accept a PR to add this functionality to the exiting plugin instead of creating a totally new plugin? The Roadie plugin already has a user base that I'm sure would like the features you are proposing, I know we would, and having this all together would be a much nicer experience.
@punkle, what's your thoughts on this from Roadie's perspective?

@awanlin I havent looked closely at the pull request, but by the looks of the screenshots, this would be an additive change to the exisiting jira plugin. On the face of it I honestly dont see why the changes couldnt be added to the roadie jira plugin.
In further support of adding it to the roadie plugin, it would reduce the support load from the main backstage/backstage project.

Hi @punkle @awanlin!
Thank you for your comments. I'll try to answer your question as best I can.
While we appreciate Roadies Jira plugin, we had vastly different goals with our Dashboard plugin. Please note that we did carry out an quite extensive analysis before starting work our own plugin, to figure out if we wanted to try to merge our features with Roadies or start fresh.
And we did came to the conclusion that the cost of merging with roadies would turn out to be expensive in efforts. So we developed our own. We actually planned to keep it internal, but a few companies have asked us to open source it. So here we are.
Some of the reasons for keeping the plugin separates includes:

We want to avoid use of proxy for querying information.
There was neither backend or frontend bits that we could reuse
If we would open source in the future, we wanted the Jira plugin to resides within the Backstage community/repository. There are both community reasons and legal reasons for this.
The cost of rewriting the existing Roadies plugin to allow integration with our features would be far higher than keeping ours as a separate dashboard focused plugin.
Since we don't share features (except interacting with Jira), it felt OK to release it as a separate plugin. You can install both of them, without gaining any redundant features.

We hope that these are good enough reasons, we don't want to step on anyone's toes. Worst case, we'll opensource it somewhere else, so the companies that have asked for the features our Jira plugin provides can still make use of the code.
Please let us know your thoughts about this :)

@punkle @awanlin I was asked to share my thoughts about this contribution by a company interested in using this plugin. I agree that it's helpful for the community to have only one plugin for Jira, but as @fridajac explained, the cost of merging plugins is none trivial. It brings substantial overhead that can be an obstacle for companies wanting to ship their Backstage portal.
@freben said he wouldn't mind having multiple home plugins and letting users choose. The features were eventually added to the home plugin in that case.
I don't want to presume that the same applies to Jira, but it could. What do you think about discussing this at an upcoming Adoption SIG meeting if we're not able to resolve it here in the near future?

@taras I don't have strong feelings about either option so I hope you don't mind me bowing out of the conversation.

Hi @fridajac @taras,
TL;DR; My intention is not to stop this but to have a dialog and come to an agreement. I like the features that are being introduced as I know it's been asked at my Org for more Jira features. It seems based on @punkle comment that Roadie is fine with this. For me the only thing I'd also like to see is thoughts from one of the Core Maintainers like @freben (and obviously the normal PR review process)
I wear many hats so this is where I'm coming from:

As a plugin author: if someone had created a brand new plugin that was very much like something I created I would be terribly frustrated if they never reached out to me to have a dialog about it and come to an agreement before hand.
As a Community member (I help many people on Discord): this creates confusion as people don't know when to use one or the other or need to use both, this is also an "expense" or "cost". I've already run into this yesterday: https://discord.com/channels/687207715902193673/1151775624541777960/1151850039338074202.
As a user of the Jira plugin: I now have to setup and install 2 plugins. We may want to use both. (Yes, for me this is trivial but it may not be for everyone using the current plugin)

The key to making a Community to work is communication, as you see it was terribly easy to ask Roadie their thoughts. Who knows maybe they would have said lets meet in the middle?

Hi!
I'm part of the Backstage team behind the Jira-plugin.

As a plugin author: if someone had created a brand new plugin that was very much like something I created I would be terribly frustrated if they never reached out to me to have a dialog about it and come to an agreement before hand.

Fair enough. As a plugin developer in quite a few different communities, I don't really see it that way, but I can absolutely appreciate that others may feel strongly about it. We really do not want to make anyone upset about this and I appreciate that you bring it up, so we can try to rectify the situation.
I can only reiterate that we did an internal investigation and come to the conclusion that our plugin would not share features, UI-components or really any code with Roadies plugin and that the cost to integrate with it was hard to motivate from a business POV at this point in time (Backstage was still on a proof-of-concept level).  We had no plan to open source it.
At a few workshop with other companies, we have been asked very kindly to open source our plugin and we figured that the feature set is so vastly different that it would be OK. It is only focused on building Jira Dashboards. No activity stream (which is the main feature of Roadies plugin).
I really, really hope that the authors of the Roadies Jira plugin is not upset or frustrated. If so, we are happy to close the pull request and maybe revisit a Roadie PR down the line when (or if) time and workload permit us to do so.
Though I want to make a few additional points when it comes to our decision to make this PR.

Roadies plugin does not resides within the Backstage monorepo, which was also important for us for both community reasons and maintainability. If Roadies plugin was a part of this repo, we may have acted differently.

Our plugin is today around 8 times larger than Roadies plugin. And a PR updating Roadies plugin would end up making us author of 95% of the code. It would be more of a replacement than a PR, which would be a bit weird too.

As a Community member (I help many people on Discord): this creates confusion as people don't know when to use one or the other or need to use both, this is also an "expense" or "cost". I've already run into this yesterday: https://discord.com/channels/687207715902193673/1151775624541777960/1151850039338074202.

As a user of the Jira plugin: I now have to setup and install 2 plugins. We may want to use both. (Yes, for me this is trivial but it may not be for everyone using the current plugin)

One is a Jira activity stream and one is a Jira dashboard plugin. While both interact with Jira (in quite different ways), I think the use cases are vastly different. We also have a separate Jira Issue Collector plugin, that allows users to submit Jira issues from Backstage, would that also be needed to be integrated in Roadies plugin? We are really trying to separate functionality into stand alone plugins. We were hopeful that would make open sourcing easier (but obviously more problematic than we counted on).

The key to making a Community to work is communication, as you see it was terribly easy to ask Roadie their thoughts. Who knows maybe they would have said lets meet in the middle?

Absolutely. And I very much regret that we did not. I don't think it necessarily would change the outcome, but it was a bit rude of me (that pushed hard to have this plugin open sourced) to not reach out beforehand. @punkle my sincere apologies for this.

Hi everyone, I'm sorry if my comments have caused any issues. My intention with my last comment was just to clarify where I was coming from and to make it clear I'm not trying to block this. Please feel free to continue moving forward, at the end of the day I am just an Org member that tries to help review PRs for my plugins and where I can so contributions can flow. Any final say will come from the Core Maintainers.
Again, apologies for any issues I may have caused. 🚀

Great work here @fridajac, code is well organized and easy to read, made giving this a review very easy. I've left a few comments, I tried to make it clear where they don't need to be handled in this PR.

Thank you! And thanks for your comments and feedback, I'll look into them :+1:

Just going to say this again but awesome work @fridajac, approving based on my review 🚀

Thank you! And thanks for all your help and feedback :)

@freben : Is this blocked by https://github.com/backstage/backstage/issues/20266?

Not in the sense that the RFC is what's been holding it back until now - I'm just slowly working through some reviews. But thanks for bringing it up though, because it became top of mind as late as today actually. Might as well start the discussion. Is there some possibility of having this entire plugin maintained off-repo? It'd bring significant benefits in terms of being unblocked from our review loop, if nothing else.

We had a discussion yesterday and decided that yes, since we will have to move out this plugin later anyway, we might just publish the plugin off-repo from the start. So I'll close this pull request. Thank you for all your help and comments @freben @awanlin @vinnie-jog-on, very appreciated, really made the plugin better :)


[WIP] Switch acm-access-token-cookie to openshift-session-token

@KevinFCormier Could you share some background for this PR? Thanks.

@clyang82 This was some investigative work for the single URL hybrid console approach. We switched course to go with dynamic plugins, so I am not sure if a common cookie name between OCP and ACM is still required. @jamestalton may have more information.


Initial commit

Description
New tool, Bug fixing, or Improvement?
Please include a summary of the change and which issue is fixed. Also include relevant motivation and context.
For new tools, please provide total size and installation time.

Related issue:
Check list

[x] Related issue / work item is attached
[x] Tests are written (if applicable)
[x] Documentation is updated (if applicable)
[x] Changes are tested and related VM images are successfully generated

https://github.com/w1s3one805/runner-images/tree/codespace-scaling-space-bassoon-4jqqrgwpg5xxh7xjq


【Bug】文章发布与更新时间以及更新时间警告的时区错误

Describe the bug
如题，设置了时区也是不行，总是在时间上加了7小时
老问题了
Reproduction
依旧如此
System Info
pnpm

Used Package Manager
pnpm
Validations

[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

I'd like to have a try at this.

根据 YAML 规范 它是预期行为，建议用目标时区的 timestamp 书写...
e.g. 2024-07-06 20:14:23 +8
是的我知道这很智障，我会转而提交一个 FAQ 条目。


Unhashable type: 'ValidateCallWrapper' when creating a model which has a @validate_call wrapped field annotation.

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
A new issue as of v2.5.0.
When creating a model:

which contains a field whose type annotation is a union of a typing.Annotated and any other type,
if a pydantic.functional_validators (BeforeValidator, AfterValidator, etc) is used within that Annotated
and the function passed to that validator has been wrapped with a validate_call decorator

then a TypeError: unhashable type: 'ValidateCallWrapper' is thrown.
If a field looks like:

name: Annotated[str, AfterValidator(validation_function)] | None it will error if that validation function is @validate_call wrapped

None can be any type.

An unwrapped validation function will not error in this case.
A type annotation which is not a Union of an Annotated with a functional_validator + validate_call decorated  and another type will not error.

The error trace  for example code is as follows:
  File "/Users/benspinks/simple.py", line 11, in <module>
    class User(BaseModel):
  File "/Users/benspinks/simple.py", line 13, in User
    name: Annotated[str, AfterValidator(lower_case_func)] | None
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
  File "/Users/benspinks/miniforge3/lib/python3.11/typing.py", line 1375, in __or__
    return Union[self, right]
           ~~~~~^^^^^^^^^^^^^
  File "/Users/benspinks/miniforge3/lib/python3.11/typing.py", line 358, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/benspinks/miniforge3/lib/python3.11/typing.py", line 481, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/benspinks/miniforge3/lib/python3.11/typing.py", line 695, in Union
    parameters = _remove_dups_flatten(parameters)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/benspinks/miniforge3/lib/python3.11/typing.py", line 326, in _remove_dups_flatten
    return tuple(_deduplicate(params))
                 ^^^^^^^^^^^^^^^^^^^^
  File "/Users/benspinks/miniforge3/lib/python3.11/typing.py", line 301, in _deduplicate
    all_params = set(params)
                 ^^^^^^^^^^^
  File "/Users/benspinks/miniforge3/lib/python3.11/typing.py", line 2151, in __hash__
    return hash((self.__origin__, self.__metadata__))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 3, in __hash__
TypeError: unhashable type: 'ValidateCallWrapper'

Example Code
from typing import Annotated

from pydantic import AfterValidator, BaseModel, validate_call

@validate_call
def lower_case_func(s: str) -> str:
    return s.lower()

class User(BaseModel):
    name: Annotated[str, AfterValidator(lower_case_func)] | None

Python, Pydantic & OS Version
pydantic version: 2.5.0
        pydantic-core version: 2.14.1
          pydantic-core build: profile=release pgo=true
                 install path: /Users/benspinks/venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.6 (main, Oct  2 2023, 13:45:54) [Clang 15.0.0 (clang-15<IP_ADDRESS>)]
                     platform: macOS-14.1-arm64-arm-64bit
             related packages: typing_extensions-4.8.0

@spinks,
Thanks a bunch for reporting this 👍. We'll look into a fix to push out with a v2.5.1 release soon. I can confirm this looks like a bug 🐛.

Addressed here: 5321fd5d


Osm filename with timestamp

Add timestamp to OSM filename, resolves issue https://github.com/onaio/OpenMapKit/issues/21

Changes in branch deprecated


Compile error

Describe your bug here.
Mind you, if you are modding, please try testing your bug in a clean version of the engine instead, this ensures the bug is actually caused by the engine itself.
I tried to compile first to edit the Source code right now but I can't Compile because of this error

Command Prompt/Terminal logs (if existing)
source/backend/animation/PsychAnimationController.hx:6: characters 16-33 : Redefinition of variable followGlobalSpeed in subclass is not allowed. Previously declared at flixel.animation.FlxAnimationController

Only this error

Are you modding a build from source or with Lua?
Source
Which branch of the engine are you using?
Latest (main)
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
No, edit nothing the source code

You may want to copy the whole code(use this)
package backend.animation;
import flixel.animation.FlxAnimationController;
class PsychAnimationController extends FlxAnimationController {
public var followGlobalSpeed:Bool = true;
public override function update(elapsed:Float):Void {
	if (_curAnim != null) {
        var speed:Float = timeScale;
        if (followGlobalSpeed) speed *= FlxG.animationTimeScale;
		_curAnim.update(elapsed * speed);
	}
	else if (_prerotated != null) {
		_prerotated.angle = _sprite.angle;
	}
}

}
If it doesn't work, you may want to update the libraries(https://github.com/ShadowMario/FNF-PsychEngine/wiki/Libraries-versions)


Implement ComputeSwiftResourceDirectory work on all platforms

Implemented a new class SwiftHost taking care of Swift directory. Made ComputeSwiftResourceDirectory work for all platforms.
@xiaobai

Did you forget to rebase by any chance?

yea, i forgot to rebase on stable. everything should be good now

@swift-ci please test

@swift-ci test and merge

@swift-ci please test Linux platform

@kusmour this seems to break linux, can you please take a look?

@kusmour this seems to break linux, can you please take a look?

will do

@swift-ci please test

@xiaobai thanks. I'll review. In the meanwhile let's run the tests.

@swift-ci please test

updated the SwiftHostTest check and replace the swift_headers_target with SWIFT_RESOURCE_PATH

@swift-ci please test

@swift-ci please test

@swift-ci please test

This is not your fault, rerunning.
/home/buildnode/jenkins/workspace/swift-lldb-PR-Linux/branch-stable/swift-corelibs-foundation/Foundation/DispatchData+DataProtocol.swift:14:8: error: missing required module '_SwiftDispatchOverlayShims'
22:12:10 import Dispatch
22:12:10

@swift-ci please test linux

@swift-ci please test

@swift-ci please test

@compnerd @dcci
All test passed, please review the change. Thank you very much :D

In the long term we should move away from custom our path variables (like LLDB_PATH_TO_SWIFT_BUILD) and towards CMake defaults, e.g. Swift_DIR that we use for find_package and the path variables this pulls in from SwiftConfig.cmake like SWIFT_BINARY_DIR.

https://github.com/apple/swift/pull/21388
I need to rebase my PR but this can't happen until that PR goes through.

I never got in touch with the C++ functions changed here. I don't know why these paths need to be computed at runtime and don't get passed in a header file via #cmakedefine? If it serves your use-case and doesn't break builds, I guess it's ok.

Not sure why we're doing that either. The goal of this PR is to move this swift-specific functionality out of lldbHost and into the SwiftExpressionParser plugin, much like the way clang works. We could change it so the directory is computed ahead of time and stuck in a header file but I think that should be a separate change.

@swift-ci please test

@swift-ci please test linux platform

@swift-ci please test

@swift-ci please test

@swift-ci please test

@swift-ci please test

I believe all the review comments have been addressed.  This also enables a bunch of additional things, so going to merge this.


Added extra callsite unit tests

test ${callsite} with a subclass. Needed for https://github.com/NLog/NLog/pull/1095

Current coverage is 72.92%

Merging #1099 into master will not affect coverage as of 89e4b52

@@            master   #1099   diff @@
======================================
  Files          263     263
  Stmts        14925   14925
  Branches      1627    1627
  Methods          0       0
======================================
  Hit          10884   10884
  Partial        418     418
  Missed        3623    3623

Review entire Coverage Diff as of 89e4b52

Uncovered Suggestions

+0.09% via ...nternal/AspHelper.cs#206...218
+0.09% via ...nternal/AspHelper.cs#188...200
+0.09% via ...nternal/AspHelper.cs#170...182
See 7 more...

Powered by Codecov. Updated on successful CI builds.


Update aida-overview.md

Remove Harness FF from this table. It was a feature that we didn't GA -- and we will be replacing it with Split capabilities.
Thanks for contributing to the Harness Developer Hub! Our code owners will review your submission.
Description

Please describe your changes: __________________________________
Jira/GitHub Issue numbers (if any): ______________________________
Preview links/images (Internal contributors only): __________________

PR lifecycle
We aim to merge PRs within one week or less, but delays happen sometimes.
If your PR is open longer than two weeks without any human activity, please tag a code owner in a comment.
PRs must meet these requirements to be merged:

[ ] Successful preview build.
[ ] Code owner review.
[ ] No merge conflicts.
[ ] Release notes/new features docs: Feature/version released to at least one prod environment.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://66cdfe34d759533f03a367a3--harness-developer.netlify.app


LOADING Redis is loading the dataset in memory

Describe the bug
When using redis-sentinel, if the slave network cannot maintain throughput to stay within the client-output-buffer limits, the slave will disconnect with the following
Predis\Response\ServerException
LOADING Redis is loading the dataset in memory

This issue has been in the library since 1.x
https://github.com/predis/predis/pull/475
The above PR provided a graceful fallback to the master in these scenarios where the slaves were temporarily unable to return read requests.  When evaluating 2.x of this library, we are still able to reproduce these scenarios resulting in exceptions rather than graceful fallback to master
Expected behavior
There are a couple of options for this scenario.

Provide a configuration option to only serve requests to the master. Load balancing is not the primary goal of using a Redis-sentinel.
Patch using a similar approach where in the catch, if the exception matches, we serve the request from the master

    private function retryCommandOnFailure(CommandInterface $command, $method)
    {
        $retries = 0;

        while ($retries <= $this->retryLimit) {
            try {
                $response = $this->getConnectionByCommand($command)->$method($command);
                break;
            } catch(\Predis\Response\ServerException $exception){

                if($exception->getMessage() == 'LOADING Redis is loading the dataset in memory'){
                    $response = $this->getMaster()->$method($command);
                }

            } catch (CommunicationException $exception) {

                $this->wipeServerList();
                $exception->getConnection()->disconnect();

                if ($retries === $this->retryLimit) {
                    throw $exception;
                }

                usleep($this->retryWait * 1000);

                ++$retries;
            }
        }

        return $response;
    }

Would a solution like this be considered? Or is there another approach recommended so that we can get this issue resolved and merged into 2.x ?
Versions (please complete the following information):

Predis: [e.g. 1.x 2.x]
PHP [e.g. 8.1.14]
Redis Server [e.g. 7.*]
OS [e.g. Ubuntu 20.04]

What do the Redis Sentinel specs say about failing commands during LOADING? I always treated it as the server being down.
Ping @chayim.

I'd say in that case let's handle in better on our end 👍
@vladvildanov Any thoughts?

@tillkruss Yeah, I like an idea to handle this on our side, also agreed that it should configurable option. The whole idea of replication is to be able to retrieve data from multiple sync sources, so it's probably should be enabled by default to switch on master on connection issues

The loading state indicates that things are backed, due to the window size on the master. The -LOADING response to ping is valid in this case.
IMHO the client shouldn't handle it, but should provide the option to the user. In some cases you want to know this is happening, and handle in on your application side. Other applications may not care - basically it depends what you're doing.
But - a retry handle could be a great answer here. This would allow the user to handle the case with their own function implementation.

@chayim
Pushing this into userland would be problematic, especially those using frameworks (for us Laravel) as we'd then be reliant on a solution being accepted/provided there.
At the very least the client should provide an option to define the read/write nodes, and not force load balancing in a redis-sentinel configuration.

So am I understanding this correctly;
Predis SentinelReplication if it gets a connection error it will retry the command but if it gets a LOADING state, it will throw the exception, while predis MasterSlaveReplication will switch to next replica or finally to the master before throwing an exception and the proposed change here is to do the same in SentinelReplication just like it is done in MasterSlaveReplication, but that was not agreed somehow? Is that right?

@mta59066
Currently the default operation of this library will load balance all read requests across the master and all read replicas. There are some checks for replica health, however p_sync state is kinda fuzzy and even thou the replica will return a LOADING state, the library will still send a request to that replica....
The PR here: #1168 allows you to force all read/writes to the master which is the work around.

@turbo124 what you proposed initially is exactly how MasterSlaveReplication works and I don't see why sacrificing on load balancing is needed to be able to handle the LOADING case on sentinel. By using SentinelReplication userland has already given that control to the client which retires the command (default 20 times). Perhaps we should just copy the retryCommandOnFailure method from MasterSlaveReplication and have them behave in the same way. Is there something I am missing between SentinelReplication and MasterSlaveReplication that should make them behave differently?

Current checks are only for sdown, not psync. that is the crux of the issue, and tbh, I am surprised upstream would even consider a node in partial resync as not sdown'd.
Another solution would be to add an additional check for psync status.


Add to diffusion tutorial

Dear Parcels community
In many lagrangian simulations, the precalculated velocity fields are used. They allow simulating the transfer of lagrangian particles due to currents (advection). However, sometimes diffusion processes are also important. Using uniform diffusion coefficients Kh is the simplest way to solve this problem. But, in the case of complex currents, the coefficient of horizontal diffusion is not uniform - Kh(i,j). There are several methods allowing calculate Kh using values of velocities and lon-lat resolutions. One of the simplest ways is using the Smagorinsky method.
I suppose to add a new example in the Diffusion tutorial describing the movement of the particles in the GlobalCurrents velocity fields for the region around South Africa taking into account the horizontal diffusion calculated by the Smagorinsky method. The corresponding example is described in the file 'example_smagdiff.py'. I think it could be useful for new Parcels users. The source code was created with the help of Daan Reijnders.

Thank you, Daan, for your contribution.


pop-up of tweet details on the map not on the right position / broken

steps:

search something on the map
click on a dot
this shows up:

You can see that there is a marker sign (with the arrow down) which overlaps with the message box. The box should be above the marker, not the marker inside.
This is one of the many totally superfluous and extremely time-killing bug issues which makes me completely depressed about this project. It would be obvious for EVERYONE that here is something broken. Instead of fixing these things right away, you just drop that stuff in the code and leave it until someone has to do MUCH MORE work to write an issue, make screen copies and put it into the workflow than it would be for you to simply change a vertical display value to make it look beautiful.
PLEASE try avoid such kind of issues in the future. PLEASE try to do complete, beautiful work.

@fuksi I think this started after the design changes regarding moving CSS files.Can you look into it?

I already noticed and fixed this four days ago at https://github.com/loklak/loklak_webclient/commit/719f55c6429cfb86e0246d9f490d2c8ca5960658
http://test.loklak.net/search?q=%23fossasia%2B%2Fmap


Can LSTM be fully quantized for inference?

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
TensorFlow installed from (source or binary): Binary
TensorFlow version (or github SHA if from source): 1.13.0-rc0

Problem Summary and Errors Encountered
I am trying to have the LSTM version from TFLite (here) quantized for inference through TFLite.
I use tf.contrib.quantize.create_training_graph() for training and have added fake quant node after the input. But I think the use of tf.unstack() creates a problem when using the 'static_rnn()`
The model is as below:
input_frequency_size = model_settings['dct_coefficient_count']
input_time_size = model_settings['spectrogram_length']
fingerprint_4d = tf.reshape(fingerprint_input,
                              [-1, input_time_size, input_frequency_size])
num_classes = model_settings['label_count']
projection_units = model_size_info[0]
LSTM_units = model_size_info[1]
with tf.name_scope("LSTM-Layer"):
  with tf.variable_scope("lstm"):
    lstm_cell = TFLiteLSTMCell(num_units=LSTM_units,
                                 use_peepholes=True,
                                 num_proj=projection_units)
    lstm_input = tf.unstack(fingerprint_4d, input_time_size, 1)
    # lstm_input = tf.fake_quant_with_min_max_args(lstm_input)
    _, last = tf.nn.static_rnn(cell=lstm_cell, inputs=lstm_input,
                                   dtype=tf.float32)
    # tf.quantization.fake_quant_with_min_max_args(last)
    flow = last[-1]

with tf.name_scope("Output-Layer"):
  W_o = tf.get_variable('W_o', shape=[projection_units, num_classes],
                          initializer=tf.contrib.layers.xavier_initializer())
  b_o = tf.get_variable('b_o', shape=[num_classes],
                          initializer=tf.constant_initializer(0.01))
  logits = tf.add(tf.matmul(flow, W_o), b_o)

After training, I freeze the graph and use the following script to convert to TF lite:
graph_def_file = "frozen_graph.pb"  # This is the .pb file.
input_arrays = ["mfcc_data"]            # This is the name of the input node
output_arrays = ["labels_softmax"]  # This is the name of the output node

# This is the main code to call tflite converter (i.e. toco)
converter = tf.lite.TFLiteConverter.from_frozen_graph(graph_def_file, input_arrays, output_arrays)
# Since fake quantization used during training, we can quantize the graph during converting
converter.inference_type = tf.lite.constants.QUANTIZED_UINT8
converter.reorder_across_fake_quant = True
input_arrays = converter.get_input_arrays()
converter.quantized_input_stats = {input_arrays[0] : (227., 1.)}  # mean, std_dev
tflite_model = converter.convert()
open("lstm-tflite.tflite", "wb").write(tflite_model)   # The resulting .tflite file.

But this then gives error as below:
ConverterError: TOCO failed. See console for info.
2019-02-06 11:57:30.492529: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 1098 operators, 1429 arrays (0 quantized)
2019-02-06 11:57:30.560084: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 1098 operators, 1429 arrays (0 quantized)
2019-02-06 11:57:30.646273: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 573 operators, 853 arrays (1 quantized)
2019-02-06 11:57:30.675022: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before pre-quantization graph transformations: 573 operators, 853 arrays (1 quantized)
2019-02-06 11:57:30.686976: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] After pre-quantization graph transformations pass 1: 522 operators, 802 arrays (1 quantized)
2019-02-06 11:57:30.703210: F tensorflow/lite/toco/tooling_util.cc:1702] Array LSTM-Layer/lstm/unstack, which is an input to the Concatenation operator producing the output array LSTM-Layer/lstm/rnn/tf_lite_lstm_cell/concat, is lacking min/max data, which is necessary for quantization. If accuracy matters, either target a non-quantized output format, or run quantized training with your model from a floating point checkpoint to change the input graph to contain min/max information. If you don't care about accuracy, you can pass --default_ranges_min= and --default_ranges_max= for easy experimentation.

I try to add the fake quant nodes manually after the unstack as you can see in the commented code above, but this will give an error from static_rnn because it will not remain a sequence anymore resulting in the following error while training:
Traceback (most recent call last):
  File "train.py", line 324, in <module>
    tf.app.run(main=main)
  File "C:\Users\bhargav\AppData\Local\Continuum\miniconda3\envs\tf113cpu\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "train.py", line 140, in main
    logits, dropout_prob = models.create_model(
  File "C:\Users\bhargav\Documents\audio\keyword-spotting\models.py", line 58, in create_model
    model_size_info, is_training)
  File "C:\Users\bhargav\Documents\audio\keyword-spotting\models.py", line 198, in create_lstm_tflite_model
    dtype=tf.float32)
  File "C:\Users\bhargav\AppData\Local\Continuum\miniconda3\envs\tf113cpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "C:\Users\bhargav\AppData\Local\Continuum\miniconda3\envs\tf113cpu\lib\site-packages\tensorflow\python\ops\rnn.py", line 1290, in static_rnn
    raise TypeError("inputs must be a sequence")
TypeError: inputs must be a sequence

I want to know if the full quantization of LSTM will be supported any time in near future. If GraphDef is needed, let me know.

Is there any update on this? Please let us know.

@aselle ptal?

Hi all,
Since it's April, I'm wondering if someone has idea on this one? I got the same issue, and it looks like I'm not alone here.
@ebrevdo @aselle

Same issue here. Any progress on quantization-aware training for LSTM/RNN?

Hi bkanaki, xddkdx, TF-deve, thanks for reaching out. We are actively working on fully quantized LSTM in the following areas:

quantization-aware training for LSTM and post-training quantization for LSTM; each of the two options can be used to produce fully quantized LSTM model, depending on one's use case.
fully quantized LSTM op that runs the model. TFLite already has a basic fully quantized LSTM (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/internal/reference/reference_ops.h#L1887) and it is being expanded to include more features such as peephole, cifg, projection etc.

Both parts are needed to fully support quantized LSTM and we are making progress. Please stay tuned.

Hi bkanaki, xddkdx, TF-deve, thanks for reaching out. We are actively working on fully quantized LSTM in the following areas:

quantization-aware training for LSTM and post-training quantization for LSTM; each of the two options can be used to produce fully quantized LSTM model, depending on one's use case.
fully quantized LSTM op that runs the model. TFLite already has a basic fully quantized LSTM (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/internal/reference/reference_ops.h#L1887) and it is being expanded to include more features such as peephole, cifg, projection etc.

Both parts are needed to fully support quantized LSTM and we are making progress. Please stay tuned.

Hi @jianlijianli , thanks for your reply! In point 1, you mentioned that post-training quantization also produces fully quantized LSTM? I thought post-training quantization still uses float32 kernels during inference (please correct me if I'm wrong). So I guess I'm not very clear about the definition of "fully quantization"? Thanks.

Hi TF-Deve, apologies for the delay. You are right that at this moment post-training quantization is using float32 activation (and uint8 weight and does quantization on the fly inside the kernel). The "fully quantized" I mentioned means a scheme that everything is integer and this is what we are currently working on.

Hello @jianlijianli , is this still "work in progress"? Because it's been 1 and a half years since this is work in progress and no updates.

Hi @jianlijianli, do you have any information on the progress of this work?

TFLite has a well documented post-training quantization path documented
here, new since I last checked this bug.
https://www.tensorflow.org/lite/performance/post_training_quantization
Have you tried that?
On Sun, Feb 21, 2021, 8:42 PM Tom Becnel<EMAIL_ADDRESS>wrote:

Hi @jianlijianli https://github.com/jianlijianli, do you have any
information on the progress of this work?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/25563#issuecomment-783075074,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANWFG7DHRHV3ZKUALDBUJ3TAHOEVANCNFSM4GUZL2RA
.

Hi @ebrevdo, thanks for responding. I have used post-training quantization and am unable to convert an LSTM using full integer quantization, although I've had no issues with dynamic range quantization of LSTM models.

hey @ebrevdo , I'm also having issues when trying to fully quantize a (keras) LSTM to int8. I'm running code like the following:
    representative_dataset_generator = build_representative_dataset_generator(
        [s.shape.as_list() for s in model.inputs],
        params,
    )

    converter = tf.lite.TFLiteConverter.from_keras_model(model)
    converter.optimizations = [tf.lite.Optimize.DEFAULT]
    converter.representative_dataset = representative_dataset_generator
    converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
    converter.inference_input_type = tf.int8
    converter.inference_output_type = tf.int8
    tflitemodel = converter.convert()

and seeing this in response:
  File "/Users/jax/code/www/tools/model_release/convert_to_tflite.py", line 137, in convert_tf2_to_tflite
    return converter.convert()
  File "/Users/jax/code/www/venv/lib/python3.7/site-packages/tensorflow/lite/python/lite.py", line 874, in convert
    self).convert(graph_def, input_tensors, output_tensors)
  File "/Users/jax/code/www/venv/lib/python3.7/site-packages/tensorflow/lite/python/lite.py", line 632, in convert
    result = self._calibrate_quantize_model(result, **flags)
  File "/Users/jax/code/www/venv/lib/python3.7/site-packages/tensorflow/lite/python/lite.py", line 449, in _calibrate_quantize_model
    calibrate_quantize = _calibrator.Calibrator(result)
  File "/Users/jax/code/www/venv/lib/python3.7/site-packages/tensorflow/lite/python/optimize/calibrator.py", line 60, in __init__
    raise ValueError("Failed to parse the model: %s." % e)
ValueError: Failed to parse the model: pybind11::init(): factory function returned nullptr.

is this a known issue? fwiw, if i comment out the lines relevant to quantization, the convert call is successful.

@liufengdb is our TFLite post-training-quantization expert so I've asked him to answer these questions.

Hi Jackson, just want to confirm that you are using the latest tf version 2.4. (Your stack shows you are still using old converter).

thanks @ebrevdo !
yes, @liufengdb , i tried both 2.4.0 and 2.4.1. is there an easy to force my local package to use the new converter?

@liufengdb :Using LSTM op in my model and could verify accuracy using TF, TFLIte float32 models.
https://colab.research.google.com/drive/1LIqNXYszOB4eUHFhqnbGLESdZs4Ipu_n?usp=sharing
However when I generate TFLite int8 model using post training quantization and int8 model doesnot work with my test data.
Note: If I use uint8 post training quantization it works fine and only for int8 quantization is not working with LSTM op

@nyadla-sys You have to put quantized output as follow this doc: https://www.tensorflow.org/lite/performance/post_training_quantization
For your case, validation code for int8 model should be:
input_scale, input_zero_point = input_details[0]["quantization"]
spectrogram = np.array(spectrogram)
spectrogram = spectrogram / input_scale + input_zero_point
q_spectrogram = np.array(spectrogram, dtype=np.int8).reshape(1,49, 257)
tflite_interpreter_quant_int8.set_tensor(input_details[0]['index'], q_spectrogram )
tflite_interpreter_quant_int8.invoke()
tflite_q_model_predictions = tflite_interpreter_quant_int8.get_tensor(output_details[0]['index'])
print(tflite_q_model_predictions)
# print("\nPrediction results shape:", tflite_q_model_predictions.shape)
# print("\nPrediction results shape:",np.array(tflite_q_model_predictions, dtype=np.int8))
output_scale, output_zero_point = output_details[0]["quantization"]
tflite_model_predictions = (np.array(tflite_q_model_predictions, dtype=np.float32) - output_zero_point) * output_scale
tflite_pred_dataframe = pd.DataFrame(tflite_model_predictions)
tflite_pred_dataframe.columns = commands
print(tflite_pred_dataframe)

It seems better than before, but I'm not so sure why uint8 and int8 result is different even if makes validation code correctly.
uint8 and int8 TFLite both uses int8 LSTM op, and I checked all tensors related to LSTM op (weights and input, by called get_tensor on python interpreter) are same but has different output. I guess TFLite kernel (unidirectional_sequence_lstm) may have some issue?

@nyadla-sys If you use
spectrogram = np.array(spectrogram)
spectrogram = np.clip(np.floor(spectrogram / input_scale + input_zero_point), 0, 255) # for uint8 validation
spectrogram = np.clip(np.floor(spectrogram / input_scale + input_zero_point), -128, 127) # for int8 validation
then int8 and uint8 model output seems almost same. Interestingly with floor and without floor makes huge difference.
Would you please check it works?

@Xhark Thanks for your time and will try as mentioned above
However I tried below for int8 model and it works for me.
spectrogram_t = np.array(spectrogram, dtype=np.uint8).reshape(1,49, 257)
spectrogram_t = np.array(spectrogram_t-128, dtype=np.int8)
tflite_interpreter_quant.set_tensor(input_details[0]['index'],spectrogram_t )

@Xhark It works and thanks

@bkanaki Could you please let us know if this issue still persists in TF v2.5 ? Thank you!

@bkanaki ,Yes, please see the below links.
https://github.com/nyadla-sys/tflite-micro/tree/micro_speech_lstm_with_training/tensorflow/lite/micro/examples/micro_speech_lstm/train
https://colab.research.google.com/github/nyadla-sys/tflite-micro/blob/upload_lstm/tensorflow/lite/micro/kernels/xtensa/examples/micro_speech_lstm/train/micro_speech_with_lstm_op.ipynb

Closing this as I have moved on to a different project a long time ago, the links above seem to be working well.

Do quantization aware training supports LSTM/GRU layers now?

https://github.com/breizhn/DTLN/issues/66#issuecomment-1312917350 with the above provided some great info on this and it would be so good to get static int8 quntisation for frameworks such as ArmNN to npu's or micro.
Does anyone know the state of play with dev for this or even an ETA?
Thanks

Do quantization aware training supports LSTM/GRU layers now?

TFMOT does not support LSTM QAT
QKeras has some kind of compatibility

Hi @JorgeRuizDev @wellido !
Have you checked fusion code lab method and inferred against int 8 quantization.
What I have observed till now that LSTM/RNN model does not work in GPU model when select ops operators are included but works in fusion code lab.
Thank you!

Do quantization aware training supports SimpleRNN layers now?


Change Font Family and Improve Navbar

Currently, the font-families used aren't so exciting for the project.  Use the appropriate combination to add some more flare to the project.
You can use tools like :

https://fontpair.co/
https://fonts.google.com/

After that design a new Navbar for the application, which satisfy the following requirements:

The navbar must need to have a white background and a box-shadow on the bottom and must stick to the top at all times.
Use appropriate colors (new-pallet) and set-up as CSS-Variables for people to use again.
The nav links on hover need to change color and add a small underline effect.
The navbar must be fully responsive on all screens.

@abhu-A-J i would like to take the issue for changing the font-families.
Can you assign that to me? Or do i have to take the entire issue.

@abhu-A-J i would like to take the issue for changing the font-families.
Can you assign that to me? Or do i have to take the entire issue.

Complete you previous issues first ! Then you may take it!


Custom keybinding UI

Need to add a UI to be able to reassign hotkeys from within the interface. Currently you must edit a json file and reload the UI to assign custom hotkeys.

Related: automatically show hotkeys when they're mapped in the relevant File / Context menus.

Added in build 1.1.1571. Can be found under the Edit menu. Shortcut to bring up the UI is ctrl+shift+b (or cmd+shift+b for macOS).


Favour reactive Maybe over Optional as a return type in ReactiveSeq

Maybe can operate either lazily and iteratively or reactively. For pull based Streams lazy + iterative works, and for push based Streams Maybe can be populated asynchronously

Merged


Contact page

This is supposed to close #59
The page is accessible at path /contact
This PR is dependent on the form-component PR #70

Header will look better once #61 is merged.

Please have a look @RishabhJain2018 @shivaniprakash95 @aka-jain @spyshiv 🍔

This is how the page will look ☝

This also contains form validation logic (added to Global Service).
This PR is dependent on #70 . Please merge that before this.

Please have a look @aka-jain 🌯
cc: @RishabhJain2018 @spyshiv

Codecov Report

Merging #71 into master will decrease coverage by 4.81%.
The diff coverage is 53.84%.

@@            Coverage Diff             @@
##           master      #71      +/-   ##
==========================================
- Coverage   64.17%   59.36%   -4.82%
==========================================
  Files           7       11       +4
  Lines         134      251     +117
  Branches        8       31      +23
==========================================
+ Hits           86      149      +63
- Misses         46       97      +51
- Partials        2        5       +3

Impacted Files
Coverage Δ

src/app/services/mock.window.service.ts
100% <100%> (ø)

src/app/global.service.ts
41.17% <11.11%> (-33.83%)
:arrow_down:

src/app/input/input.component.ts
44.82% <44.82%> (ø)

src/app/services/window.service.ts
50% <50%> (ø)

src/app/contact/contact.component.ts
76.66% <76.66%> (ø)

Impacted Files
Coverage Δ

src/app/services/mock.window.service.ts
100% <100%> (ø)

src/app/global.service.ts
41.17% <11.11%> (-33.83%)
:arrow_down:

src/app/input/input.component.ts
44.82% <44.82%> (ø)

src/app/services/window.service.ts
50% <50%> (ø)

src/app/contact/contact.component.ts
76.66% <76.66%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db9b77a...ce7b1e0. Read the comment docs.

Added the API request to this. Please merge #70 before this.
This also contains common code for the API service which will be required by other pages as well. Please review it soon 🍕
cc: @aka-jain @spyshiv @RishabhJain2018 @shivaniprakash95 @deshraj

Thanks for the links @RishabhJain2018 😄 I made the changes you suggested.
Please have a look 🔢

@live-wire Can you please check why the travis build is failing.

@live-wire I think we should also display the toast notification to the user saying some like we have received your query and will reply soon.
Please refer to the current version of EvalAI.

@RishabhJain2018 I added a TODO for that in this PR.
Custom Toasts are a part of #77 and the toast you mentioned is added there 😄

@live-wire More conflicts :D

I'll resolve these right after #66 is merged 💃

@live-wire Please resolve the merge conflicts here.

On it!

Closing this since it is already merged.


[Feature] EventMesh integrate with milvus as log broker

Search before asking

[X] I had searched in the issues and found no similar issues.

Feature Request
EventMesh can be the log broker for the milvus, eventmesh sdk can be provided for integration.
https://github.com/milvus-io/milvus/issues/25127
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Interested and I can look into this

@RSTdefg Hi! May I kindly inquire about the current progress? Your update would be greatly appreciated. Thank you!


[zh] Sync architecture/garbage-collection.md

Sync with en text:
content/zh-cn/docs/concepts/architecture/garbage-collection.md

/approve


Move to Google OAuth for verification

Following discussion in the server, I think replacing the current email verification system with Google OAuth is the right way forward. Whether we use multiple OAuth or not is a point of contention, but for now keeping user experience in mind, we can send an ephemeral link message to a web application.
This basically requires removing the Modal logic, and just have one persistent view. This should be implemented without having to use the raw on_interaction event which Discord has refused to acknowledge the drawbacks of.
I'm self assigning this for now, and will get to it next week depending on my other workload. In the meantime, if someone wants to pick this up, feel free to. However please mention this issue in your commit messages/PR content so that I am kept in the loop.

/assign me

What if I am willing to check my email & its spam folder. And I want to use email based verification?

you might be, but the majority of people find the email thing annoying
and oauth2 is more elegant and secured and gives us infrastructure for potential interaction with the google API should the need arise. the decision to switch to OAuth over email has already been finalized.
If you have any objections based on concrete examples where it provides a worse user experience or similar, feel free to elaborate.

I don't have any objection. I was just suggesting to not completely purge the email system.
Most of the websites I have seen haven't purged the email sign up option in favor OAuth. And I think we too shouldn't.

these are all examples of what we are not. we don't allow signing up with random emails. we want to verify an email provided by Google, hence OAuth.
as i said this discussion has already taken place and implementations are underway. you can bring up any concerns you have in the server.

closing as this has been sniped.


Update to v0.211.2 🚧

DO NOT MERGE!!
I am still testing this update for the issue @sagararyal mentioned in https://github.com/sarveshpro/n8n-heroku/pull/58#issue-1549076429
I tried running the branch on a fresh instance and it works fine. So I am guessing it is some issue in the migration command.
@nosovk do you have any insights on this issue as you are also running n8n on production.

closing this as this issue seems to be related to few versions from 206 - 215. Migration works fine on 216


Failed to load the Ensime package - At Duplicate key 'atom-text-editor'

Ensime fails to load after installation.
**Atom Version**: 1.12.2
**Electron Version**: 1.3.6
**System**: Mac OS X 10.11.6
**Thrown From**: [Ensime](https://github.com/ensime/ensime-atom) package, v0.39.0

Stack Trace
Failed to load the Ensime package

At Duplicate key 'atom-text-editor'

At Duplicate key 'atom-text-editor'
  at Object.detectDuplicateKeys (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/season/lib/cson.js:268:13)
  at /Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:115:38
  at Array.reduce (native)
  at Obj (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:107:30)
  at transformNode (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:197:12)
  at Value (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:79:14)
  at transformNode (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:197:12)
  at Block (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:76:14)
  at transformNode (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:197:12)
  at Object.parse (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/cson-parser/lib/parse.js:203:12)
  at parseObject (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/season/lib/cson.js:42:23)
  at parseContentsSync (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/season/lib/cson.js:68:16)
  at Object.readFileSync (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/node_modules/season/lib/cson.js:170:14)
  at /Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:387:38
  at Array.map (native)
  at Package.module.exports.Package.loadKeymaps (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:384:46)
  at /Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:110:19
  at Package.module.exports.Package.measure (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:92:15)
  at Package.module.exports.Package.load (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:106:12)
  at PackageManager.module.exports.PackageManager.loadPackage (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package-manager.js:466:14)
  at /Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package-manager.js:410:19
  at Config.module.exports.Config.transact (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/config.js:312:16)
  at PackageManager.module.exports.PackageManager.loadPackages (/Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/package-manager.js:405:19)
  at /Users/abhijit_sarkar/Applications/Atom.app/Contents/Resources/app.asar/src/atom-environment.js:772:28

Commands

Config
{
  "core": {
    "disabledPackages": [
      "language-toml",
      "language-sql",
      "language-sass",
      "language-ruby-on-rails",
      "language-ruby",
      "language-python",
      "language-php",
      "language-property-list",
      "language-perl",
      "language-objective-c",
      "language-mustache",
      "language-make",
      "language-less",
      "language-javascript",
      "language-go",
      "language-css",
      "language-csharp",
      "language-coffee-script",
      "language-clojure",
      "language-c",
      "dev-live-reload",
      "autocomplete-html",
      "autocomplete-css",
      "language-text"
    ],
    "projectHome": "/Users/abhijit_sarkar/Workspace/atom",
    "telemetryConsent": "no",
    "warnOnLargeFileLimit": 10
  }
}

Installed Packages
# User
browser-plus, v0.0.82 (inactive)
Ensime, v0.39.0 (inactive)
hyperclick, v0.0.39 (inactive)
intentions, v1.1.2 (inactive)
language-scala, v1.1.8 (inactive)
linter, v1.11.18 (inactive)
minimap, v4.25.6 (inactive)
atom-dark-syntax, v0.27.0 (inactive)
atom-dark-ui, v0.52.0 (inactive)
atom-light-syntax, v0.28.0 (inactive)
atom-light-ui, v0.45.0 (inactive)
base16-tomorrow-dark-theme, v1.3.0 (inactive)
base16-tomorrow-light-theme, v1.3.0 (inactive)
one-dark-ui, v1.6.2 (inactive)
one-light-ui, v1.6.2 (inactive)
one-dark-syntax, v1.5.0 (inactive)
one-light-syntax, v1.5.0 (inactive)
solarized-dark-syntax, v1.0.5 (inactive)
solarized-light-syntax, v1.0.5 (inactive)
about, v1.7.0 (inactive)
archive-view, v0.62.0 (inactive)
autocomplete-atom-api, v0.10.0 (inactive)
autocomplete-css, v0.13.1 (inactive)
autocomplete-html, v0.7.2 (inactive)
autocomplete-plus, v2.31.4 (inactive)
autocomplete-snippets, v1.11.0 (inactive)
autoflow, v0.27.0 (inactive)
autosave, v0.23.1 (inactive)
background-tips, v0.26.1 (inactive)
bookmarks, v0.42.0 (inactive)
bracket-matcher, v0.82.2 (inactive)
command-palette, v0.39.0 (inactive)
deprecation-cop, v0.54.1 (inactive)
dev-live-reload, v0.47.0 (inactive)
encoding-selector, v0.22.0 (inactive)
exception-reporting, v0.40.0 (inactive)
find-and-replace, v0.202.2 (inactive)
fuzzy-finder, v1.4.0 (inactive)
git-diff, v1.1.0 (inactive)
go-to-line, v0.31.0 (inactive)
grammar-selector, v0.48.2 (inactive)
image-view, v0.60.0 (inactive)
incompatible-packages, v0.26.1 (inactive)
keybinding-resolver, v0.35.0 (inactive)
line-ending-selector, v0.5.0 (inactive)
link, v0.31.2 (inactive)
markdown-preview, v0.158.8 (inactive)
metrics, v1.1.1 (inactive)
notifications, v0.65.1 (inactive)
open-on-github, v1.2.1 (inactive)
package-generator, v1.0.1 (inactive)
settings-view, v0.243.1 (inactive)
snippets, v1.0.4 (inactive)
spell-check, v0.68.4 (inactive)
status-bar, v1.4.1 (inactive)
styleguide, v0.47.2 (inactive)
symbols-view, v0.113.1 (inactive)
tabs, v0.103.0 (inactive)
timecop, v0.33.2 (inactive)
tree-view, v0.210.0 (inactive)
update-package-dependencies, v0.10.0 (inactive)
welcome, v0.35.1 (inactive)
whitespace, v0.35.0 (inactive)
wrap-guide, v0.38.2 (inactive)
language-c, v0.54.0 (inactive)
language-clojure, v0.22.1 (inactive)
language-coffee-script, v0.48.0 (inactive)
language-csharp, v0.12.1 (inactive)
language-css, v0.40.1 (inactive)
language-gfm, v0.88.0 (inactive)
language-git, v0.15.0 (inactive)
language-go, v0.43.0 (inactive)
language-html, v0.46.1 (inactive)
language-hyperlink, v0.16.1 (inactive)
language-java, v0.24.0 (inactive)
language-javascript, v0.122.0 (inactive)
language-json, v0.18.3 (inactive)
language-less, v0.29.6 (inactive)
language-make, v0.22.2 (inactive)
language-mustache, v0.13.0 (inactive)
language-objective-c, v0.15.1 (inactive)
language-perl, v0.37.0 (inactive)
language-php, v0.37.3 (inactive)
language-property-list, v0.8.0 (inactive)
language-python, v0.45.1 (inactive)
language-ruby, v0.70.2 (inactive)
language-ruby-on-rails, v0.25.1 (inactive)
language-sass, v0.57.0 (inactive)
language-shellscript, v0.23.0 (inactive)
language-source, v0.9.0 (inactive)
language-sql, v0.25.0 (inactive)
language-text, v0.7.1 (inactive)
language-todo, v0.29.1 (inactive)
language-toml, v0.18.1 (inactive)
language-xml, v0.34.12 (inactive)
language-yaml, v0.27.1 (inactive)

# Dev
No dev packages

Couldn't reproduce but there definitely is a duplicate entry for atom-text-editor in the config.cson. Don't know why you have harder validation of this than I do though. Fastest way to verify fix is probably if you just edit the file according to the change here:
https://github.com/hedefalk/ensime-atom/commit/26c6bf0140e3efebeaa6eeb308c988e6aea888b7
in your local installation to see if that fixes the issue. Thanks!

Thanks @hedefalk. That fixed it, although I moved things around just a little bit.
'atom-text-editor':
  'f4': 'ensime:go-to-definition'
  'cmd-ctrl-i': 'ensime:show-implicits'
  'shift-cmd-t': 'ensime:search-public-symbol'
  'ctrl-alt-d': 'ensime:go-to-doc'
  'ctrl-alt-D': 'ensime:browse-doc'
'atom-text-editor[data-grammar="source scala"]':
  'f2': 'ensime:lock-type-hover'

Came here to say I got the same issue.
It happened on my 2011 MBP on macOS Sierra (both on existing install, and after a fresh Atom install). However, it is OK on 2015 MBP with El Capitan.
Some clarification the steps required to fix it:

Go to ~/.atom/packages/Ensime/keymaps
Edit ensime.cson to match the above (essentially, merge the two entries under 'atom-text-editor')

Below is the original ensime.cson which cause the issue (as a reference):
'atom-text-editor':
  'f4': 'ensime:go-to-definition'

'atom-text-editor[data-grammar="source scala"]':
  'f2': 'ensime:lock-type-hover'

'atom-text-editor':
  'cmd-ctrl-i': 'ensime:show-implicits'
  'shift-cmd-t': 'ensime:search-public-symbol'
  'ctrl-alt-d': 'ensime:go-to-doc'
  'ctrl-alt-D': 'ensime:browse-doc'

@hhandoko Thanks for your info!
I'll re-open this until the change is released (which it unfortunately isn't yet)

I have the same issue as well.

I have the same issue (although its on windows 10 this time), with a fresh install.

same issue here as well. I'm using Ubuntu 16.04

confirm the issue on lastest macos sierra

I also have this issue

Same issue (Windows 10)

Same issue on updated Ubuntu 16.10 , on a freshly installed Atom ( installed by .deb ).
So it should be easy to reproduce the bug.

Same issue, fresh install on macOS Sierra.

ensime.cson proposed by @asarkar seems to work for me. Linux Mint 17.3 Rosa.
Slightly changed "cmd" to "alt".


Remove empty claimable balances check

I think this isn't behaving how we expect and seems like it will be empty for channels when it is still viable. I don't think this optimization did much either way.

what determines when we stop watching it then? Because something should stop watching old channels.

I think ldk will internally, this just makes sure we don't prematurely stop

Please confirm that in some way. There's plenty of things LDK should do that we can't assume they actually do.

this didn't fix @futurepaul's issue so just going to close


blueprint, template and credential could be deleted before usage generation

Please @keyki review it!

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/cloudbreak-pull-request/802/

Please do not merge!

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/cloudbreak-pull-request/810/

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/cloudbreak-pull-request/815/

Necessary db changes:

ALTER TABLE template ADD COLUMN deleted boolean DEFAULT false;

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/cloudbreak-pull-request/822/

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/cloudbreak-pull-request/832/

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/cloudbreak-pull-request/845/

It's your turn.


python311Packages.boschshcpy: 0.2.56 -> 0.2.57

Diff: https://github.com/tschamm/boschshcpy/compare/0.2.56...0.2.57
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 230306 run on x86_64-linux 1

5 packages built:

home-assistant-component-tests.bosch_shc
python310Packages.boschshcpy
python310Packages.boschshcpy.dist
python311Packages.boschshcpy
python311Packages.boschshcpy.dist


Specify file name when there is an unknown/corrupt image format

I tried to convert a big directory with sub directories.

It would be great to have the file name of the problematic image in the error message

Wow ! I hope you have enough memory.
Indeed, yes, this happen in a multi thread env. I will improve the error message (may be close the processing).
Also may be useful to add an "ignore-errors" options, to keep going but just display warnings.

done for improve part of the error message
checkout last version:
go install<EMAIL_ADDRESS>

Thanks!
The "corrupt" file was in webp format.
I created a new PR

#6


Update to newer versions of dependencies.

Coverage decreased (-0.4%) to 61.97% when pulling c5a25f18cb796b5624eb3acf45e881012674a692 on deflaux:master into 28398ceaa79a38b04d1562513b8467ca36a4ed71 on googlegenomics:master.

Coverage decreased (-0.4%) to 61.97% when pulling 00ddfc3cd17968e14d1af212a25f866df8333b6e on deflaux:master into 28398ceaa79a38b04d1562513b8467ca36a4ed71 on googlegenomics:master.

Coverage decreased (-0.4%) to 61.97% when pulling b3985a60c1df05da0fca758e57060306d2d25fce on deflaux:master into 28398ceaa79a38b04d1562513b8467ca36a4ed71 on googlegenomics:master.


Enable all strict type checking options.

🚀 Feature request
Command (mark with an x)

- [x] new
- [ ] build
- [ ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Description
set compilerOptions strict:true in tsconfig.json

Are there plans to enable all strict type checking options? @alan-agius4

We'll discuss and see if I'd want to enable this for default projects. We'll post the outcome of the discussion here.

@xiaoxiangmoe, thanks for opening this. We'll very much like to do this, but at the same time we want to ensure that the generated code by the schematics works with this new option.

@alan-agius4 What do I need to do to help you?

Well, you can take this task and implement it :) It's free for grab.

Anyone mind if I take this one?

@dkosasih https://github.com/angular/angular-cli/pull/14905

@alan-agius4 I'm very happy this is finally happening. I proposed that a long time ago and did several PRs, but they were never accepted.
Unfortunately, except I'm missing something, the PR is incomplete as the option has not been added in the ng new schematics, so ng new --strict is failing.

It's also missing the strictFunctionTypes flag.


Inference on nuscenes dataset

Hi,
I want to use the deflow module to calculate the flow between two frames of point cloud of nuscenes dataset (2Hz, since the keyframes with annotations are at 2Hz). I have get the computed flow, but I found the result is not correct, is it because 2Hz is too slow?
The parameters used are:
voxel_size = [0.2, 0.2, 8]  (actually, voxel_size  0.1 and 0.4 are tried, but the result is still not correct
grid_feature_size = [512, 512]
point_cloud_range = [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]
num_iters = 2 / 4
Thank you, looking forward to your reply.

The pre-trained model is trained on 10Hz Argoverse 2. If you infer on nus with 10Hz (which I tried), it's still good enough without any training.

The pre-trained model is trained on 10Hz Argoverse 2. If you infer on nus with 10Hz (which I tried), it's still good enough without any training.

Thank you very much! I will try 10Hz nuscenes data.

I don't know what is going wrong... The red one is pc0, the green one is pc1, and the blue one is pc0+flows. It seems that the flow is also added to the places that should be considered static. In the image above, where there are dynamic points, the flow is going foward to the right place, but it is not yet completely aligned with pc1.

I think I need your code for processing nus data to tell the reason..

Thank you, the main code is shown below.
In class Nuscenes_pt(data.Dataset):
def getitem(self, index):
info = self.nusc_infos[index] # for pc1
lidar_sd_token = self.nusc.get('sample', info['token'])['data']['LIDAR_TOP'] # for pc1
    # 获取前两帧sweep的lidar_sd_token_prev(10Hz间隔)
    prev_token = self.nusc.get('sample_data', lidar_sd_token)['prev']
    prev_prev_token = self.nusc.get('sample_data', prev_token)['prev'] # for pc0

    # 获取lidar_channel
    lidar_token = lidar_sd_token
    lidar_token_prev = prev_prev_token
    lidar_channel = self.nusc.get("sample_data", lidar_token)
    lidar_channel_prev = self.nusc.get("sample_data", lidar_token_prev)

    if self.version == "v1.0-trainval":
        lidar_path = info['lidar_path'][16:]
        lidar_path_prev = lidar_channel_prev['filename']
    elif self.version == "v1.0-mini":
        lidar_path = info['lidar_path'][44:]
        lidar_path_prev = lidar_channel_prev['filename']
        # lidar_path_prev = info_prev['lidar_path'][44:]
    elif self.version == "v1.0-test":
        lidar_path = info['lidar_path'][16:]
        lidar_path_prev = lidar_channel_prev['filename']

    points = np.fromfile(os.path.join(self.data_path, lidar_path), dtype=np.float32, count=-1).reshape([-1, 5])
    points_prev = np.fromfile(os.path.join(self.data_path, lidar_path_prev), dtype=np.float32, count=-1).reshape([-1, 5]) # N*5

    # t-1时刻点云转换到t时刻点云 for flow部分
    pcd_trans_flow_prev = PCDTransformTool(points_prev[:, :3])
    # First step: transform the t-1 pointcloud to the ego vehicle frame; t-1 lidar→ t-1 ego
    cs_record_prev = self.nusc.get('calibrated_sensor', lidar_channel_prev['calibrated_sensor_token'])
    pcd_trans_flow_prev.rotate(Quaternion(cs_record_prev['rotation']).rotation_matrix)
    pcd_trans_flow_prev.translate(np.array(cs_record_prev['translation']))

    # Second step: transform from ego to the global frame at timestamp of the first frame; t-1 ego→0ego
    poserecord_prev = self.nusc.get('ego_pose', lidar_channel_prev['ego_pose_token'])
    pcd_trans_flow_prev.rotate(Quaternion(poserecord_prev['rotation']).rotation_matrix)
    pcd_trans_flow_prev.translate(np.array(poserecord_prev['translation']))

    # Third step: transform from global first frame to t pointcloud ego; 0ego→t ego
    poserecord = self.nusc.get('ego_pose', lidar_channel['ego_pose_token'])
    pcd_trans_flow_prev.translate(-np.array(poserecord['translation']))
    pcd_trans_flow_prev.rotate(Quaternion(poserecord['rotation']).rotation_matrix.T)

    # Fourth step: transform from ego into the lidar. t ego→t lidar
    cs_record = self.nusc.get('calibrated_sensor', lidar_channel['calibrated_sensor_token'])
    pcd_trans_flow_prev.translate(-np.array(cs_record['translation']))
    pcd_trans_flow_prev.rotate(Quaternion(cs_record['rotation']).rotation_matrix.T)
    points_prev_transtot = pcd_trans_flow_prev.pcd.T

    # 记录Fego
    pose_1 = get_lidar_pose(self.nusc, lidar_channel)
    pose_2 = get_lidar_pose(self.nusc, lidar_channel_prev)
    # 计算自车运动矩阵 Fego
    Fego = np.dot(np.linalg.inv(pose_2), pose_1)

The code above is the data reading and preprocessing part, which has converted the pc0 point cloud to the pc1 coordinate system. After this part, 'points_prev_transtot' is input to deflow module as pc0, 'points' is input as pc1.
Then, I use the deflow module to align pc0 and pc1.
####################################################
Deflow pointcloud alignmenet (Deflow Module)
####################################################
pc0 = fusion_dict['return_xyz_prev'] # torch.Size([34752, 3]) fusion_dict['return_xyz_prev'] is 'points_prev_transtot'
pc0_before_pseudoimages, pc0_voxel_infos_lst = self.embedder(pc0) # pc0_voxel_infos_lst:torch.Size([32201, 3])
pc1 = fusion_dict['xyz_cart'] # torch.Size([34784, 3]) fusion_dict['xyz_cart'] is 'points'
pc1_before_pseudoimages, pc1_voxel_infos_lst = self.embedder(pc1) # pc1_voxel_infos_lst[0]['points'].shape: torch.Size([32186, 3])
grid_flow_pseudoimage = self.backbone(pc0_before_pseudoimages, pc1_before_pseudoimages) # ([1, 64, 512, 512])
flows = self.head(
torch.cat((pc0_before_pseudoimages, pc1_before_pseudoimages), dim=1),
grid_flow_pseudoimage, pc0_voxel_infos_lst)
and the deflow parameters are set in init function:
num_iters = 2
voxel_size = [0.2, 0.2, 8] # 0.2 0.2
grid_feature_size = [512, 512] # 512 512
point_cloud_range = [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]
self.embedder = DynamicEmbedder(voxel_size=voxel_size, pseudo_image_dims=grid_feature_size, point_cloud_range=point_cloud_range, feat_channels=32)
self.backbone = FastFlow3DUNet()
self.head = ConvGRUDecoder(num_iters = num_iters)
the functions  DynamicEmbedder, FastFlow3DUNet(), ConvGRUDecoder are all from deflow/scripts/network folder

Besides, the visualization code is as follows:
可视化pc0+flows 和 pc1 看分别对应点坐标
    allpoint_pc0 = o3d.geometry.PointCloud()
    allpoint_pc0.points = o3d.utility.Vector3dVector(pc0[0].cpu().numpy())
    allpoint_pc0.paint_uniform_color([1, 0, 0])  # 红色
    allpoint_pc1 = o3d.geometry.PointCloud()
    allpoint_pc1.points = o3d.utility.Vector3dVector(pc1[0].cpu().numpy())
    allpoint_pc1.paint_uniform_color([0, 1, 0])  # 绿色
    o3d.io.write_point_cloud("pcd_pc0_all.ply", allpoint_pc0)
    o3d.io.write_point_cloud("pcd_pc1_all.ply", allpoint_pc1)

    pc0_flow_points = pc0_voxel_infos_lst[0]['points']
    pc0_newpoints = pc0_flow_points + flows[0]
    pc0_newpoints_np = pc0_newpoints.detach().cpu().numpy()

    pcd_pc0 = o3d.geometry.PointCloud()
    pcd_pc0.points = o3d.utility.Vector3dVector(pc0_newpoints_np)
    pcd_pc0.paint_uniform_color([0, 0, 1])  # 蓝色
    # 保存点云和中心点到文件
    o3d.io.write_point_cloud("pcd_pc0_flow.ply", pcd_pc0)

Is it because only two frames are used so the flow is not accurate？

Is there any the smallest test script that I can run fast (one script to export h5py file so I can run model by my self)? Otherwise it's hard to get your problem here. You can reference this h5py format we need here: https://github.com/KTH-RPL/DeFlow/blob/main/dataprocess/README.md#process

Here is one of deflow_best model result in 10Hz nus. (I only downloaded mini set, the figure I show here from: scene_id: scene-0103, timestamp:<PHONE_NUMBER>551488)

I can upload the check_flow vis func in the test/scene_flow.py script.

Thanks a lot! TAT I really appreciate your time and work. I'll try to get the h5py format script and use the check_flow vis func.

I noticed that there is a ground point filtering process in the data preprocessing part (extract_av2.py), but the nus data doesn‘’t have real-valued ground height like av2 to directly filter out the ground points. How did you filter ground points when testing on nus data? Did you use the Z threshold directly (for example, set Z<=-1.5 to be the ground point)? Besides, I have some trouble on writing extract_nus.py like extract_av2.py. Can I reference your code for testing nus data?
And I may found the reason why the code I used didn't work before. I didn’t see the preprocessing code (extract_av2.py) at first. I only combined the paper and the core scripts/network code. I directly transferred the pc0 point cloud to the lidar coordinate system of pc1, and used the converted pc0 and original pc1 directly as the input of the deflow module without going through the data preprocessing part like extract_av2 (such as calculating pose_flow caused by ego-car motion and ground point filtering).

For nuscene, I directly use my own Python func linefit to do ground segmentation: https://github.com/Kin-Zhang/linefit. While DeFlow Paper Section IV-A said we use only non-ground points as input here: Leveraging HD maps provided by most autonomous driv- ing datasets [32], [11], or by deploying ground segmentation techniques like [33], we can easily exclude ground points from both Pt and Pt+1.
Of course, you can directly try z<=-1.5 to filter ground for quick test.
And for the script, as said in README: https://github.com/KTH-RPL/DeFlow/blob/main/dataprocess/README.md#dataset It needs time with work under review to publish waymo and nus data script.
Since in DeFlow's paper, we only focus on av2 data. And the scripts we are talking about now are for other private papers.
Let me know after you try inputting without a ground point into DeFlow network.

The above is the result after removing the ground points. The result of these two frames look better than before, but when zoomed in, the flow of the ego vehicle and the dynamic vehicle seems to be not right. I wonder if it is still a problem of nus data preprocessing.
I want to confirm about the input point cloud. Is pc0 converted to the pc1 lidar coordinate system and then input together with pc1 itself, and finally the flow generated by the network is added to the converted pc0 point cloud as the visualization result?
Besides, I also tried to debug in the original code to compare the original preprocessing process with the code I used, and to see where is different. But I found that there was a problem with the path.
In extract_av2.py:
def main(
    argo_dir: str = "../data/av2",
    output_dir: str ="../data/av2/preprocess",
    av2_type: str = "sensor",
    data_mode: str = "test",
    mask_dir: str = "../data/av2/3d_scene_flow",
    nproc: int = (multiprocessing.cpu_count() - 1)
):
    data_root_ = Path(argo_dir) / av2_type/ data_mode
    output_dir_ = Path(output_dir) / av2_type / data_mode
    output_dir_.mkdir(exist_ok=True, parents=True)
    process_logs(data_root_, output_dir_, nproc)
    create_reading_index(output_dir_)

I set the argo_dir, output_dir as above, but there is an error: [Errno 2] No such file or directory: '../data/av2/sensor/train/00a6ffc1-6ce9-3bc3-a060-6006e9893a1a/map/../data/av2/sensor/train/00a6ffc1-6ce9-3bc3-a060-6006e9893a1a/map/log_map_archive_00a6ffc1-6ce9-3bc3-a060-6006e9893a1a____PIT_city_31785.json'
It can be seen that the first half of the path is repeated twice, but I didn't find where the path was joined during the debugging process. Do you know what is wrong?
Thanks a lot, looking forward to your reply~

Thanks, I have the file but the path in the code is not right TAT.... I dont know why the part "../data/av2/sensor/train/00a6ffc1-6ce9-3bc3-a060-6006e9893a1a/map/" is repeated twice in the path

Could you try the absolute path with full path, I tried the code and run again just now, I cannot reproduce your result.
python dataprocess/extract_av2.py --av2_type sensor --data_mode train --argo_dir /home/kin/data/av2 --output_dir /home/kin/data/av2/preprocess

Thanks a lot!

I have tried on av2 dataset, and the generated flow is right (have run the extract_av2.py, 3_vis.py and scene_flow.py). But when I tried on nus data, the flow is not correct. I think maybe is the problem of the input point cloud. I found the corresponding two frames of point cloud near scene_id: scene-0103, timestamp:<PHONE_NUMBER>551488, which is timestamp:<PHONE_NUMBER>448916 as pc0 and timestamp:<PHONE_NUMBER>551488 as pc1.
For quick test, I directly put the two frames of point cloud in deflow.py and got the flow. The test code added in deflow.py is:
```
# test nus
ply_file_path = "/opt/data/private/DeFlow/tests/pcd_pc0_all.ply"
pc0 = o3d.io.read_point_cloud(ply_file_path)
points = np.asarray(pc0.points)
point_pc0s = torch.tensor(points, dtype=torch.float32)
point_pc0s = point_pc0s.unsqueeze(0)
pc0_before_pseudoimages, pc0_voxel_infos_lst = self.embedder(point_pc0s)
    ply_file_path = "/opt/data/private/DeFlow/tests/pcd_pc1_all.ply"
    pc1 = o3d.io.read_point_cloud(ply_file_path)
    points = np.asarray(pc1.points)
    point_pc1s = torch.tensor(points, dtype=torch.float32)
    point_pc1s  = point_pc1s.unsqueeze(0)
    pc1_before_pseudoimages, pc1_voxel_infos_lst = self.embedder(point_pc1s)

    grid_flow_pseudoimage = self.backbone(pc0_before_pseudoimages,
                                        pc1_before_pseudoimages)

    device = grid_flow_pseudoimage.device # cpu add----- this part is for device unification
    pc0_before_pseudoimages = pc0_before_pseudoimages.to(device)
    pc1_before_pseudoimages = pc1_before_pseudoimages.to(device)

    flows = self.head(
        torch.cat((pc0_before_pseudoimages, pc1_before_pseudoimages),
                dim=1), grid_flow_pseudoimage, pc0_voxel_infos_lst)

    pc0_valid_point_idxes = [e["point_idxes"] for e in pc0_voxel_infos_lst]
    pcd3 = o3d.geometry.PointCloud()
    pcd3.points = o3d.utility.Vector3dVector((point_pc0s[0,:,:][pc0_valid_point_idxes[0]]+flows[0]))
    # pcd2.points = o3d.utility.Vector3dVector(pc0[:, :3] + data[flow_mode][~gm0])
    pcd3.paint_uniform_color([0.0, 0.0, 1.0]) # blue: pc0 + flow
    o3d.io.write_point_cloud("nus_result.ply", pcd3)

In this way, the result after added flows is almost the same as pc0. But the result you got is correct. Do you know what causes the flow to be almost the same as pc0? Attached files are the pc0 and pc1 files I got after processing and input into the deflow module. Are they the same as the ones you input into the network?
[pc0 and pc1.zip](https://github.com/user-attachments/files/16146145/pc0.and.pc1.zip)

AHa, Thanks for providing the detail of this, I knew the reason now:
It's because of the flow we predict is deltaF. (check the eq. 2 in our paper) and I also attached code here for you to aware that we made a transform based on the pose1 and pose2 you provided.

https://github.com/KTH-RPL/DeFlow/blob/2e567945aa3d1cf107c06683ec2c27b7eac68c0d/scripts/network/models/deflow.py#L57-L74
By the way, I update the linefit (ground segmentation) config for nuscene if you want to use:

https://github.com/Kin-Zhang/linefit/blob/master/assets/config/nus.toml
https://github.com/Kin-Zhang/linefit/blob/master/assets/config/README.md

Let me know if you have more questions.

Thanks for providing the linefit config for nuscene~
I added the pose_flows to the result, that is, transform_pc0 + flows(from the network) + pose_flows is the final visualization reuslt. (Does this match your idea？) But I found the final result is still different with yours.

Can I have your pose0to1 matrix between the two frames mentioned previously (timestamp:<PHONE_NUMBER>448916 as pc0 and timestamp:<PHONE_NUMBER>551488 as pc1) ? My pose_0to1 is:

The translation in Y in my pose_0to1 is 0.85m, but the (transform_pc0 + flows from network) and the transform_pc0 taht I got have more than 1m in tranlsation in Y.  So I wonder if there is something wrong with my calculated of pose_0to1.

Hi, I checked my ego_pose with pose0 and pose1 for you to check:

And the transform_pc0 means pc0 -> pc1 which contains pose_flow inside now. And network estimate \delta F. The final flow in the h5 file is because we will save output+pose_flow = final_flow inside the key, attach code here:
https://github.com/KTH-RPL/DeFlow/blob/a71c9ef454c48b92a521dcd3f928b1e47da92328/scripts/pl_model.py#L249-L266

Thanks a lot! I tried transform_pc0 + flows(from the network) + pose_flows  and the flow in the dynamic object (car) is correct. But I found a new question, the points of static objects (such as buildings) in transform_pc0 (original pc0 transformed to pc1 coordinate system) are the same as in pc1, in this case, the result adding pose_flows seems to be wrong. But in the figure you commented on Jun 26, the points of buildings in pc0 (in red) are in different location compared with the points in pc1. Is pc0 in the figure the transform_pc0 ?


Enable pip install vanna, pip install vanna[snowflake], pip install vanna[bigquery]

We'll likely have to begin using setuptools for this

If we're going to support more and more databases, it's going to take forever (and also possibly cause dependency issues) if we begin installing every database driver under the sun when you do pip install vanna
The standard pattern for this is to do bracketed installation of database drivers individually as needed