[designspaceLib] Add lib key for name table name ID 25 (Variations PostScript Name Prefix)

Variable fonts may need a name ID 25 (e.g. to avoid this bug in Photoshop CC 2019: https://feedback.photoshop.com/photoshop_family/topics/photoshop-cc-2019-variable-fonts-with-separate-fonts-for-uprights-and-italics-snap-back-to-upright-when-changing-an-axis). See https://docs.microsoft.com/en-us/typography/opentype/spec/name#name-ids.
It can be put into the default font's openTypeNameRecords, but if you have multiple Designspaces referencing different subsets of the same set of UFOs, they might need different name ID 25s. Instances also don't need it. The field should therefore be settable in the Designspace, in something like a public.openTypeVariationsPostScriptNamePrefix lib key.

Yes, I think we should define a public.openTypeVariationsPostScriptNamePrefix DS lib key for that. Ufo2ft should set it accordingly after having generated the VF.

@josh-hadley told me that Adobe apps expect a unique ID 25 per distinct (VF) file. So the output filename stem (filtered according to spec) could be the default value. Or the DSv5 VF name.

There is a similar bug in InDesign 18.1 (Windows) as described in the initial comment. It happens only if the fonts are installed in the Windows system Fonts-Folder. It does not happen if the fonts are installed in the InDesign fonts folder.
Please define a public.openTypeVariationsPostScriptNamePrefix DS lib key for that.


Add bringup property to swarming task properties for query purpose

The bringup property in .ci.yaml controls whether we should treat a test as flaky.
It would be useful to promote it to properties in the swarming task bigquery, so that we can easily view analytics excluding bringup: true ones, namely the flaky tests.

https://flutter-review.googlesource.com/c/infra/+/18502 to add.


Add getAttributes support for Android

Description

This pull request addresses the issue described here: https://github.com/wix/Detox/issues/2083

In this pull request, I have introduced support for attributes for android including for View, TextView and Checkbox.

For features/enhancements:

[x] I have added/updated the relevant references in the documentation files.

For API changes:

[ ] I have made the necessary changes in the types index file.

Kudos, @jonathanmos!
@d4vidi , WDYT , when can we take this into processing?

@noomorph of course!


Use processing_class instead of tokenizer in LogCompletionsCallback

What does this PR do?
Fixes #2260
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a GitHub issue? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines.
[ ] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


[DO NOT MERGE] SaslPlainSslEndToEndAuthorizationTest

Changes needed to get SaslPlainSslEndToEndAuthorizationTest to pass in KRAFT

Removed direct calls in EndToEndAuthorizationTest to zookeeper and instead
use the superuser principal to update the Acls through the brokers. This
removes use of deprecated featuer --authorizer-properties

Hi @pprovenzano . Thanks for the PR.
Before I can review this I'd like to understand the strategy here. Do you want to leave the CONTROLLER endpoint as PLAINTEXT? Or will it also be SASL? If it is plaintext, then our job will be simpler here, but we are testing less, of course. In general the main focus of the test is on brokers doing authorization, so perhaps the controller being plaintext is OK?
As you noticed, there are some bootstrapping calls to AclCommand. What are we replacing the calls to AclCommand with? Are we going to be modifying the Authorizer objects directly? That is what we have done in other tests. We will need to modify all of the Authorizer objects though (each broker has one)

The main focus of these tests is on brokers doing authorization and validating that the client gets (or does not get) access to the topic for various ACLs. So I think that we should not bother to update the CONTROLLER to use something other than PLAINTEXT. That said, for SaslPlainSslEndToEndAuthorizationTest you still need to set a principal so that BROKERS can update the CONTROLLER.
What I have done is set a super user principal which is different from the client and that principal calls the brokers to set ACLs. I believe the BROKER just forwards the request to zookeeper or a CONTROLLER.

Rather than putting the @Timeout(60) on each test case, I'd suggest putting it on the test class itself, so that it applies to each test.
Actually, can you split out a separate PR for just that change? It would be a good first change for you.


[Bug]: outbound http requests TIMEOUT

Description
I am self hosting my Remix App on a Hetzner cloud server. I am not able to make a call to an external API endpoin, the resend API in this case.
I have a firewall config that include all the ports mentioned by Coolify docs.
All outbound traffic is allowed.
This is the error I get from the log.
sendEmail FetchError: request to https://api.resend.com/emails failed, reason: connect ETIMEDOUT <IP_ADDRESS>:443
    at ClientRequest.<anonymous> (/app/node_modules/.pnpm/@remix-run+web-fetch@4.4.2/node_modules/@remix-run/web-fetch/dist/lib.node.cjs:1800:11)
    at ClientRequest.emit (node:events:532:35)
    at TLSSocket.socketErrorListener (node:_http_client:502:9)
    at TLSSocket.emit (node:events:520:28)
    at emitErrorNT (node:internal/streams/destroy:170:8)
    at emitErrorCloseNT (node:internal/streams/destroy:129:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  erroredSysCall: 'connect'
}

I already contacted the resend team and they told that they are not blocking any traffic from hetzner.
I would appreciate any help here, I am aware that this may not be a coolify problem also.
Thanks again for building such an amazing product
Minimal Reproduction (if possible, example repository)

Buy a hetzner server
install Coolify v4.0.0-beta.297
create a new project
deploy a remix app to the created project
use nixpack for building
in the app make a call to the resend API

Exception or Error
sendEmail FetchError: request to https://api.resend.com/emails failed, reason: connect ETIMEDOUT <IP_ADDRESS>:443
at ClientRequest. (/app/node_modules/.pnpm/@remix-run+web-fetch@4.4.2/node_modules/@remix-run/web-fetch/dist/lib.node.cjs:1800:11)
at ClientRequest.emit (node:events:532:35)
at TLSSocket.socketErrorListener (node:_http_client:502:9)
at TLSSocket.emit (node:events:520:28)
at emitErrorNT (node:internal/streams/destroy:170:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
type: 'system',
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
erroredSysCall: 'connect'
}
Version
v4.0.0-beta.297

This issue is not related to Coolify itself.
Can you please execute the following commands on your server and show me the output?

curl ``[https://api.resend.com/emails](https://api.resend.com/emails)
docker exec coolify sh -c "curl -s ``[https://api.resend.com/emails](https://api.resend.com/emails)``"

@andrasbacsai for some reason now everything works fine 🎉  It's indeed not coolify itself. Apparently was an issue on the resend side.
Appreciate your help and thanks again for the amazing product

Some more update for folks having a similar issue.
From the resend team:

The connectivity problems are due to sanctions or abusive behaviours associated with certain IPs/ranges from Hetzner that have been flagged and blocked by major cloud providers. Unfortunately, we do not have control over this.

There's a few reports on forums and blog posts about this situation.
As a workaround, I recommend rotating the public IP of your application to ensure it falls within a different range and try to communicate with our API again.
For my case I did not need to do that and the ip was unblocked


Loading Overlay doesn't work with spinner/animate

I am using React 16.2 with Redux 3.7.2.  I get an error using this component but only when setting spinner or animate to true.
Here is the basic structure of my component:
import React from 'react';
import Loadable from 'react-loading-overlay';

class MyComponent extends React.Component {
    constructor(props) {
        super(props);
    }

    render() {
    	<div>
	    	<Loadable
			active={this.props.loading}
			spinner
			text="Loading..."
		>
			<MySubComponent .../>
	    	</Loadable>
		...
	</div>
    }
}

Here is the error:
app.bundle.js:14462 Warning: React.createElement: type is invalid -- expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.

Check the render method of `LoadingOverlayWrapper`.
    in LoadingOverlayWrapper (created by MyComponent)
    in div (created by MyComponent)
    ...

Uncaught Error: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.

My guess is FirstChild is returning null/undefined in the CSSTransitionGroup here: https://github.com/derrickpelletier/react-loading-overlay/blob/master/src/LoadingOverlay.js#L39
Please advise.

Any progress on this? I have the same issue

This is resolved i believe, sorry for the delay.
It's fixed in v1.0.0.


Add autogen-modules

Silly question: what are autogen-modules?

The right way to supress
<no location info>: warning: [-Wmissing-home-modules]
    Modules are not listed in command line: Build_doctests

but not break cabal sdist (it will be broken, if Cabal isn't >= 2.0, i.e. you aren't using GHC-8.2)

Cabal-2.0 said to me
setup: Error: Could not find module: Build_doctests with any suffix:
["gc","chs","hsc","x","y","ly","cpphs","hs","lhs","hsig","lhsig"]. If the
module is autogenerated it should be added to 'autogen-modules'.

Ah, OK! I assume that older Cabals will simply not recognize the autogen-modules stanza and emit a warning?

Yes, except it seems it breaks with GHC-8.0 :/


Return SDP products from CMR instead of generating fresh when they are requested

waiting on swot-sds for CRID-selection and internal discussion on SDP selection.


Config file option to provide IG profiles

HI!
Would be great to load a file as "profiles.txt" with all the needed profiles to be downloaded right there. Actually, the workaround
$ python pyinstastories.py -d `cat profiles.txt`

should work, but maybe is not the more elegant solution.
What do you think?
Thanks!

This has now been added in https://github.com/notcammy/PyInstaStories/commit/b44c7a203d2b9fc74d561fd485db924a5262de42. 🙂

Great job! ;)


Query builder should add components rather than replacing them. Example: Calling WHERE multiple times should add additional clauses, rather than replace the only clause

Describe the bug
The query builder replaces rather than adds components and clauses.
This goes against the behavior of every single query builder I've ever worked with, and is rather surprising and unintuitive, while also making some situations difficult or impossible.
Environment (please complete the following information):

OS: linux and mac
Database: postgres
Database driver: pgx v5
Jet version: 2.11.1

Code snippet
func (d DAO) SelectAllAccountsByFilter(ctx context.Context, filters models.Filters) ([]model.Accounts, error) {
	query := SELECT(
		Accounts.AllColumns,
	).FROM(
		Accounts,
	)

	if len(filters.Names) > 0 {
		query = query.WHERE(Accounts.Name.IN(Strings(filters.Names)...))
	}
	if filters.Active != nil {
		query = query.WHERE(Accounts.Active.EQ(Bool(*filters.Active)))
	}
	if len(filters.FavColors) > 0 {
		query = query.WHERE(Accounts.FavColor.IN(Strings(filters.FavColors)...))
	}

Expected behavior
When passed a Filters struct with all three values set, I expected to see an SQL statement that included 3 clauses in the WHERE portion.
Instead, only the last one was set.
I expect this kind of "builder" behavior for all parts of the query, including the SELECT, the FROM, the ORDER_BY, everything.  (I should be able to call SELECT multiple times, to add new columns to be selected. Same for FROM, ORDER_BY, everything, etc.)
Why? Because building complex queries or queries for complex api's, are often dependent on logic, so there is often a need to change parts of the query based if statements.
Yes, I realize an ugly workaround would be to build it separately, then pass it into JET when finished, but besides being ugly, it defeats a lot of the purpose of a query builder.

What you describe as expected behavior is not an SQL builder but an ORM. SQL builder tries to mimics the underling sql without hiding any query details.
For instance:
	if len(filters.Names) > 0 {
		query = query.WHERE(Accounts.Name.IN(Strings(filters.Names)...))
	}
	if filters.Active != nil {
		query = query.WHERE(Accounts.Active.EQ(Bool(*filters.Active)))
	}

ORM will add silently AND between these two conditions, but sql builder should not. My only regret is not adding a panic if WHERE or any statement clause is called twice.
SQL builder tries to map golang DLS code to sql, one to one. Since sql SELECT query can have only one WHERE clause, sql builder SELECT statement should have only one WHERE clause.
condition := Bool(true)

if len(filters.Names) > 0 {
	condition = condition.AND(Accounts.Name.IN(Strings(filters.Names)...))
}
if filters.Active != nil {
	condition = condition.AND(Accounts.Active.EQ(Bool(*filters.Active)))
}
if len(filters.FavColors) > 0 {
	condition = condition.AND(Accounts.FavColor.IN(Strings(filters.FavColors)...))
}

query := SELECT(
	Accounts.AllColumns,
).FROM(
	Accounts,
).WHERE(
	condition,
)

Yes, I realize an ugly workaround would be to build it separately, then pass it into JET when finished, but besides being ugly, it defeats a lot of the purpose of a query builder.

Couldn't disagree more. When you are calling a go function, do you first call a function, and then modify it parameters, or you first calculate parameters and then call a function. It seems to me you spend to much time using GORM.

I have never used GORM, and I dislike ORM's quite a bit for many reasons.
I would consider an SQL builder to be something that builds an SQL statement.
Here, lets take github.com/Masterminds/squirrel for a try.  It is a SQL builder, and that is all it does:
func (d DAO) SelectAllAccountsByFilter(ctx context.Context, filters models.Filters) ([]models.Account, error) {
	query := sq.
		Select(
			"id",
			"name",
			"email",
			"active",
			"fav_color",
			"fav_numbers",
			"properties",
			"created_at").
		From("accounts").
		OrderBy("id")

	if len(filters.Names) > 0 {
		query = query.Where(sq.Eq{"name": filters.Names})
	}
	if filters.Active != nil {
		query = query.Where(sq.Eq{"active": *filters.Active})
	}
	if len(filters.FavColors) > 0 {
		query = query.Where(sq.Eq{"fav_color": filters.FavColors})
	}

	sqlStr, args, err := query.PlaceholderFormat(sq.Dollar).ToSql()
	if err != nil {
		return nil, err
	}

The result is as expected, if there are multiple filters, each of them will be added onto the WHERE statement.
This is one of the main purposes of an SQL builder, to let you build out an SQL statement following business logic that often requires you to add to the SQL based on the results of if statements.
If I knew exactly what the SQL should look like, without any if statements involved, then I would just write straight SQL myself without any need for involvement by an SQL builder.  (I often already do that everywhere there isn't any logic involved.)
Having to "build" the SQL statement myself manually, before passing it into Jet's "sql builder", seems like a big miss.
But speaking practically, maybe there is an opportunity for something like a WHERE_AND function.

If you want to quote the SQL builder library, at least quote the real SQL builder library. For example, this is JOOQ for Java:
public Condition condition(HttpServletRequest request) {
    Condition result = noCondition();

    if (request.getParameter("title") != null)
        result = result.and(BOOK.TITLE.like("%" + request.getParameter("title") + "%"));

    if (request.getParameter("author") != null)
        result = result.and(BOOK.AUTHOR_ID.in(
            select(AUTHOR.ID).from(AUTHOR).where(
                    AUTHOR.FIRST_NAME.like("%" + request.getParameter("author") + "%")
                .or(AUTHOR.LAST_NAME .like("%" + request.getParameter("author") + "%"))
            )
        ));

    return result;
}

// And then:
create.select()
      .from(BOOK)
      .where(condition(httpRequest))
      .fetch();

By the looks of it, `squirrel' is more of an ORM than a SQL builder.

The result is as expected, if there are multiple filters, each of them will be added onto the WHERE statement.

Probably for the ORM users.

But speaking practically, maybe there is an opportunity for something like a WHERE_AND function.

For start WHERE_AND is not an SQL keyword. In some cases, it is worth adding a new statement method if there is a clear benefit. For example, the MODEL method of insert statement for type safety benefits. But, adding a new method just to support SQL builder anti-pattern programming is a big NO.

Squirrel is probably the most popular Golang SQL Builder library.  It is not an ORM at all.

I don't think I would call squirrel a complete SQL Builder. Except for some basic sql keywords(like select, where, order by...), everything else is still basically a raw string. Meaning the risk of sql injection attacks is only slightly reduced compared to raw string.
Regarding WHERE_AND you can get something similar using global AND function:
conditions := []BoolExpression{Bool(true)}

if len(filters.Names) > 0 {
	conditions = append(conditions, Accounts.Name.IN(Strings(filters.Names)...))
}
if filters.Active != nil {
	conditions = append(conditions, Accounts.Active.EQ(Bool(*filters.Active)))
}
if len(filters.FavColors) > 0 {
	conditions = append(conditions, Accounts.FavColor.IN(Strings(filters.FavColors)...))
}

query := SELECT(
	Accounts.AllColumns,
).FROM(
	Accounts,
).WHERE(AND(
	conditions...,
    ),
)

If squirrel always adds AND after each Where, it means it is impossible to completely replace the where condition during SQL building phase. That feels like a limiting factor.

Squirrel leads to some truly unmaintainable code, was not a good experience in production.
I'm with jet on this one, though I can see where you're coming from.
Squirrel is more sql generation using strings with go operations.
We're targeting a transparent type safe way to build sql queries and scan them into native objects. Rather than simply utilizing go operations, go-jet takes a more go idiomatic / philosophical approach to sql generation.
ie. you can't write WHERE condition WHERE condition in SQL so you shouldn't be able to write it in Jet.

Unrelated and not a bash at squirrel but it seems to take all of the cons of SQL and Jet and mash them together, I hope we don't transition in that direction. Breaking up raw sql into go operations for the sake of it makes little sense. The areas where squirrel shines makes for unmaintainable code as it's too complex to reason or makes a "don't touch this" query.
Squirrel chops up a string into go operations. It's not really fluent. And doesn't seem to serve much purpose for this tradeoff. As a "feature" you can do multiple .Where() and write raw sql again, which segments out a raw strings that are partial sql but also violate SQL. Worst of all worlds which I'll never understand.
SQL native.
SELECT * FROM elephants WHERE name IN ('Dumbo', 'Verna') AND age > 5;

Squirrel which chops sql strings and lets you violate sql as practice?
psql.Select("*").
    From("elephants").
    Where("name IN (?,?)", "Dumbo", "Verna").
    Where("age > ?", 5).
    ToSql()

Jet which doesn't chop strings and mimics SQL 1:1 + type safety (the reason for not writing sql)
SELECT(table.Elephants.AllColumns).
FROM(table.Elephants).
WHERE(table.Elephants.Name.In(names...).
    AND(table.Elephants.Age.GT(5)

Well, I guess I'll just leave it with this: It would be nice to have a WHERE_AND() function, ideally one that can be called without having to call WHERE first, so that you can put it in if/switch statements.


Use mono_repo v2, merge smoke_test jobs

Did it change how we should run the mono_repo tool? Can you check it here?
https://github.com/dart-lang/pub-dartlang-dart/blob/master/doc/development.md#working-with-mono_repo

Did it change how we should run the mono_repo tool? Can you check it here?
https://github.com/dart-lang/pub-dartlang-dart/blob/master/doc/development.md#working-with-mono_repo

Updated @isoos


add a README

fixes #4

I will add an example about how to launch the test with the env vars.
Otherwise, I try to add some comment to improve the work you have done until now.


fix irsa composition passing config provider name

Signed-off-by: Carlos Santana<EMAIL_ADDRESS>What does this PR do?
fix irsa composition passing config provider name
Motivation
The composition for irsa is broken due to not passing the config provider name from the claim down to managed resources of type Role. When is not passed the default value for config provider name is default and if user didn't name the config provider default the managed resource fails
More

[x] Yes, I have tested the PR using my local account setup  (Provide any test evidence report under Additional Notes)

[ ] Yes, I have added a new example under examples to support my PR

[ ] Yes, I have updated the docs for this feature

[ ] Yes, I have linked to an issue or feature request (applicable to PRs that solves a bug or a feature request)

Note:

Not all the PRs require examples and docs
We prefer small, well tested pull requests. Please ensure your pull requests are self-contained, and commits are squashed

For Moderators

[ ] E2E Test successfully complete before merge?

Additional Notes

This was addressed in https://github.com/awslabs/crossplane-on-eks/pull/67. Sorry about this, it was my oversight. Thanks for this though!


Handle pending charges

currently will count a charge  that is pending (e.g. a bank debit) as paid, when it is still in transit


Ported Text Escape/ Unescape tool

Pull request type

Please check the type of change your PR introduces:

[ ] Bugfix
[X] New feature or enhancement
[ ] UI change (please include screenshot!)
[ ] Code style update (formatting, renaming)
[X] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Internationalization and localization
[ ] Other (please describe):

What is the current behavior?
DevToys 2.0 doesn't have a Text Escape / Unescape tool yet
Issue Number: #468
What is the new behavior?
Ported the tool from DevToys 1.0 to 2.0. Issue #468 is solved through the fact that DevToys 2.0 has an option to select what end of line character should be used in Settings. Let's start with this for now. If unhappy, we could create an option specifically in this tool.
Other information

Quality check
Before creating this PR:

[X] Did you follow the code style guideline as described in CONTRIBUTING.md
[X] Did you build the app and test your changes?
[ ] Did you check for accessibility? On Windows, you can use Accessibility Insights for this.
[ ] Did you verify that the change work in Release build configuration
[X] Did you verify that all unit tests pass
[X] If necessary and if possible, did you verify your changes on:

[X] Windows
[ ] macOS (DevToys 2.0)
[ ] Linux (DevToys 2.0)

Hey @btiteux

Before merging this pull request #971 , we have one to add support for Linux escape lines, too. Can we add the support at the same time?

As I said in the PR description, issue #468 (which #971 solves in DevToys 1.0) is solved through the fact that DevToys 2.0 has an option to select what end of line character should be used in Settings. I'd suggest to limit ourselves to this for now and wait for more feedback. The reason of this push back is that the end of line sequence is set at the Monaco editor level. So replacing end of line in the string we hold in memory won't be efficient because Monaco will overwrite it constantly (when the user do any edit, or when copying or pasting into the editor).
A workaround is to add a new API to DevToys.API that would allow to customize the end of line sequence for a given IUIMultiLineTextInput. I'd like to try to avoid it (for now) in order to keep the API simple and don't give too much freedom (which translates into a risk of inconsistency between tools).
The setting I'm referring to above has already been tested on Linux.

Note that \r (Mac) is not supported by Monaco. Note also that \r on Mac is not supported after Mac OS 9, which DevToys doesn't target anyway.

we need some Unit Tests on the GUI :)

Done :-)

We can go with the settings for the moment. But I think it's more user friendly to have a list here and that would avoid changing the settings for the whole apps.

We can go with the settings for the moment. But I think it's more user friendly to have a list here and that would avoid changing the settings for the whole apps.

I ultimately agree :-)


freeciv-2.5.5_1: Failed reading graphics file

Installed freeciv via brew install homebrew/games/freeciv and upon completion, I attempted to launch the game with freeciv-gtk2.  This unfortunately failed with the following message:
2: Didn't find the option file. Creating a new one.
0: Failed reading graphics file "/usr/local/Cellar/freeciv/2.5.5_1/share/freeciv/misc/treaty.png": "Couldn't recognize the image file format for file '/usr/local/Cellar/freeciv/2.5.5_1/share/freeciv/misc/treaty.png'"

That seems strange. Might be related with outdated dependencies. It has no problem with graphics loading on my local machine.
Would you follow the steps in the troubleshooting guide and post the details if you still have the problem? Thanks!

Now we have a new version (2.5.6) in the tap (#756). Please let us know if you still encounter any issue.


SkeletonIK works incorrectly in Godot 3.3

Godot version:
3.3
OS/device including version:
Win10, PC, GLES3, Nvidia Geforce RTX2070
Issue description:
I opened my 3.2.3 project in 3.3 and noticed that the SkeletonIK node behaves erratically.
I had its target set to the camera's rotation so that the player's torso moves with the camera, which works perfectly fine in 3.2.3, but in 3.3 the SkeletonIK seems to be ignoring its target and instead looking in erratic directions (it's impossible to find the pattern here).
Steps to reproduce:
Import a skeleton, set a SkeletonIK with a target, set the target to copy the rotation of the camera.
Minimal reproduction project:
don't have one:(

Can confirm, the previously working IK setup is now twisted

Yes, something got messed up around RC5. Suggest returning to the RC4 commit for SkeletonIK - I believe it worked fine until RC5. Heard it also works OK in 4.0 which apparently is using the previous code for SkeletonIK (haven't tried 4.0).

Can't reproduce your issue with my test Skeletons as all bones rotate as expected for me in Godot 3.3.
This issue needs a reproduction project with your Skeleton setup.
From your description I expect the bug fixes in #47738 for root bone rotation and solver update to be the issue for your skeleton setup no longer working as before.

CC @TwistedTwigleg @AndreaCatania

Can't reproduce your issue with my test Skeletons as all bones rotate as expected for me in Godot 3.3.
This issue needs a reproduction project with your Skeleton setup.
From your description I expect the bug fixes in #47738 for root bone rotation and solver update to be the issue for your skeleton setup no longer working as before.

The project is huge and messy - it would be difficult to extract just the minimum reproduction component. But I've recorded this bug, take a look (maybe that gives you a hint?):
https://www.youtube.com/watch?v=titXIneUxIc

The project is huge and messy - it would be difficult to extract just the minimum reproduction component. But I've recorded this bug, take a look (maybe that gives you a hint?):
https://www.youtube.com/watch?v=titXIneUxIc

If you cannot create a minimal working example, could you make your project downloadable, at least for devs who want to try to debug this?
While a video is good, nothing beats a (non-)working example, because using that a develop can find the exact commit that broke it (instead of say "it broke between RC4 and RC5" you can see that commit abcdef broke it) and it can also confirm that the fix actually works.

@macryc
I don't need your entire project. Just your skeleton resource (any maybe skinned mesh for visuals) and your SkeletonIK setup saved in a packedscene should be enough to debug this.

Out of curiosity, is it only the bone twist that is the issue? Prior to #47738 the root/first bone wouldn’t rotate itself correctly. If all the rotation, not just the twist, is incorrect then #47738 may have not fixed the issue in this case.

It looks like the bone is twisting. Overall the posture seems correct - it works the same way it did in 3.2.3. It's the root bode (the 1st spine bone, in this case) that is rotating incorrectly. The tip and the bone in the middle seem to be working fine, otherwise the character wouldn't be looking at the target mark (I'm referring to the video).

I managed to get your Player.zip working (was broken on load) and removed dependencies and everything that is unnecessary for this issue.
here is a new MRP for debug
Player.zip
Your skeleton has an inverted spine bone rest pose but this alone is not the problem. Your inverted bone gets twisted when SkeletonIK kicks in and rotates the bone to -180 degree. I think the problem is that parts of the SkeletonIK source are made with Euler rotations while in a sane world we could use Quat's to prevent such rotation errors.

... I think the problem is that parts of the SkeletonIK source are made with Euler rotations while in a sane world we could use Quat's to prevent such rotation errors.

I’m not sure Euler angle use is the direct problem. Whether you are using Euler or Quat, the rotation of a bone to a target position has infinite possibilities, so you’d have to resolve using some method. How you resolve that rotation is where it makes a bit of a difference, but more from an implementation perspective than an end result.
(Unless you are interpolating, then yeah, Quat rotation is MUCH better 🙂)
My hunch is that the issue is caused by using look_at to generate the Basis for rotation, and the calculation there is leading to -180 degrees of twist rotation rather than reaching the same point using swing rotation. I wrote the code to use look_at for the root bone because otherwise the BoneAttachment3D node wouldn’t work when assigned to the root bone.
That said, I have a hunch on how to revert the code so the root bone is rotated like the other bones while also having BoneAttachment3D nodes working, I just haven’t had the time to try it. I think that executing the IK code after _process will fix the issue with BoneAttachment3D nodes while still using the old code, but the code in SkeletonIK will need some minor adjusting to get there. I plan to look at it soon, probably this weekend assuming nothing unexpected happens with my schedule.

I managed to get your Player.zip working (was broken on load) and removed dependencies and everything that is unnecessary for this issue.
here is a new MRP for debug
Player.zip
Your skeleton has an inverted spine bone rest pose but this alone is not the problem. Your inverted bone gets twisted when SkeletonIK kicks in and rotates the bone to -180 degree. I think the problem is that parts of the SkeletonIK source are made with Euler rotations while in a sane world we could use Quat's to prevent such rotation errors.

Thanks mate. Yea, I just posted the player resource on its own - should have mentioned there's bound to be tons of dependencies to ignore/remove. This is a Mixamo skeleton. I did think that there may be some issues with the original rest pose of the spine bone but it looked fine in Blender (perhaps something to do with the importer). Anyway, like you said, that isn't the issue. I'm curious to see what you find. Thanks for looking at it.

Hi everyone, I have the same issue with a Mixamo skeleton. Here is the minimal project: https://github.com/Naurk/TP-Minimal-Controller
Just apply a SkeletonIK and you will see the twisted root bone, this didn't happen in 3.2.3 and previous. Sorry if I'm redundant, probably it depends on orientation issues, but do you know how can be solved (preferably directly in Godot)?

Hi everyone, I have the same issue with a Mixamo skeleton. Here is the minimal project: https://github.com/Naurk/TP-Minimal-Controller
Just apply a SkeletonIK and you will see the twisted root bone, this didn't happen in 3.2.3 and previous. Sorry if I'm redundant, probably it depends on orientation issues, but do you know how can be solved (preferably directly in Godot)?

TwistedTwigleg has already fixed it. There is a request to pull the fix into the next release (3.3.1., I believe). Just need to wait a bit.

The PR @macryc mentioned is this one: https://github.com/godotengine/godot/pull/48166

Apart from turning, there's a problem with scaling the bone. Godot version: v3.2.3 / 3.3
It was the №2 problem in my report, but the topic was closed.
https://github.com/godotengine/godot/issues/48358

Godot and blender file.
Ik_test.zip

@okay300 - are you scaling the bone, the Skeleton, or the SkeletonIK node? Also, was this issue occurring prior to Godot 3.2.3?

The defect appears if you use SkeletonIK along with zoom animation.   I zoom in through the animation of Bone 1.

Godot and blender file: Ik_test.zip

Can I recreate a theme that touches bone scaling?

@okay300 - thank you for the clarification on the issue and additional pictures! Sorry I didn't get back sooner, things have been busy on my end.
Can you test the artifact Godot editor builds here (Windows link) to see if they fix the scaling issue? The PR still has an issue with interpolation that I need to fix (planning to try and tackle it this weekend). You can find the build by clicking the "Artifacts" button in the top right corner of the screen and then downloading the editor.
In my testing the issue was fixed in that build, but if you can still reproduce it, then it is probably a separate issue, in which case we can reopen the issue you made previously and/or make a new issue.

Now animation does not zoom bone 1 heirs.
Testing in fixed version - Fixes the SkeletonIK

if you scale in a godo, scaling works.
godo and blender project.zip

I tested the above project in the latest version of #48251 (updated yesterday) and I can confirm that there still appears to be an issue with the scaling animation when IK is playing.


Add 'writable' DataArray accessor

Description
I was thinking on adding this in a discussion point but it takes the same effort to make it in a draft PR and, if we all agree, developed it further into a proper PR. If not, we just close the PR.
Based on the conversations today, it seems advisable to protect the Data object from accidental writing given that all the DataArrays it contains are always passed by reference and modifying them somewhere will affect the calculations everywhere else.
This PRs makes all DataArrays added to the Data object read-only and implements a DataArray accessor that doubles as context manager and makes the chosen DataArray (any DataArray, not just those in the Data object) temporarily writable. It requires explicitly setting the context manager for the particular array to modify a requirement in order to modify it, reducing the chances - although not eliminating completely - of things being modified where they should not.
Fixes # (issue)
Type of change

[ ] New feature (non-breaking change which adds functionality)
[ ] Optimization (back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist

[ ] Make sure you've run the pre-commit checks: $ pre-commit run -a
[ ] All tests pass: $ poetry run pytest

Further checks

[ ] Code is commented, particularly in hard-to-understand areas
[ ] Tests added that prove fix is effective or that feature works

Looks useful! Were you also thinking of adding a decorator that can be used to do the same thing?

You would need as input the DataArray you want to make writeable, so I'm not sure how this could be used in decorator form.
Besides, what would be the use case of having it in decorator form?

Quick answers:

There's no comments, docstrings or tests, and I bypassed pre-commit because this is meant to be a discussion. That's what's also a Draft PR. If you give me the green light, I'll put all those things in place.
Flags are a numpy feature, not a xarray feature, that's why they are applied to the data attribute of the DataArray, which is the underlying numpy array.
For this reason, it obviously does not prevent a whole replacement of the DataArray - that brings a whole new numpy array to the table. If you want to prevent that, to some extend, put measures in place in the __set__ method of the Data object to prevent overwriting an existing DataArray.
This measure prevents you from making changes to entries when you should not (as it happened to @jacobcook1995 and as it was a concern for @vgro), but the real benefit is precisely for the future developers who will not have the broad vision and awareness that you all have now and can easily make those mistakes. So, if anything, I see this as something more useful in the future than now. Therefore, for me a safemode mode does not make sense - it defeats the purpose of setting this in the first place.

Aha - I hadn't fully got that it was setting the attribute on the ndarray via data.  I also wasn't expecting it to prevent (3) - just noting that this is a limitation.
On point 4 - yes, I agree that this is a future-proofing feature that helps developers get their models right. And that is really valuable, particularly because data being updated at the wrong time is a particularly evil kind of bug to track down: the predictions are wrong, but no errors are occurring. I think that has a huge amount of value. My concern is that it's a bit like training wheels on a bike: great to make sure you don't fall off to begin with, but eventually they start to get in the way. This PR would insist that all models use training wheels all the time.
I may be over-estimating the computational cost, but writing data is going to be a common activity. If a model has internal fast time scales, then it might be updating a lot and that could build up to be an appreciable cost.  In an ideal world, we'd like to be able to remove the context manager wrapping for trusted models, and that is why I mentioned a safe mode.
I've been trying to think of ways that we could allow us to take those wheels off for trusted models, but I can't see a good approach to do it at any level.

There doesn't seem to be much point in having a safe mode inside of the context manager. Once the update code is wrapped inside a context manager block then those special methods (__init__, __enter__, __exit__) have to run and there isn't a great deal to shortcut.

The only way to avoid running the context manager is to take the code outside of the manager, but  we definitely don't want something like this all over the place:
if SAFEMODE:
    with T.writable:
        T[1] = 0
else:
    T[1] = 0

We could have a set of  'privileged' module names, but there is no easy way that the numpy array can know which module is writing to it. I think that the module identity would have to be passed at the level of individual attempts to edit:
# Trusted module
T[1] = (0, 'trusted_module')

# Untrusted module
with T.writable:
    T[1] = 0

This is genuinely closer to what I could see us wanting. Write the model using the context manager and then swap out the context managers once the model is trusted, but it seems clumsy and a really unintuitive use of '='.

Setting the writable flag as a global option simply doesn't work:
if SAFEMODE:
    self.data[key].data.setflags(write=False)

When SAFEMODE = False then modules can use a mix of direct writing and context manager wrapped writing,  which does allow wheels to be taken off. But when SAFEMODE = True, all the trusted modules fail.

You would need as input the DataArray you want to make writeable, so I'm not sure how this could be used in decorator form.
Besides, what would be the use case of having it in decorator form?

Ahh, good point. I hadn't thought of that. I only mentioned it because I thought it came up in our discussion. But using a with block is probably clearer in any case.

Of course it's an error to write data inefficiently! This PR wouldn't necessarily stop that, it would just be written inefficiently within a context manager.
All I'm saying is that this PR introduces an unavoidable layer in our data API whose sole purpose is to make people stop and think that they are doing the right thing. And I think that's valuable, but I'm concerned that this layer can't easily be removed in production code.

Right, but what I'm saying is how frequently are you realistically going to be using this context manager? For example, if you only needed to use it once a second, the code of setting and unsetting a flag is so negligible then it's not worth worrying about. And if it's 10,000 times a second, then you're doing something odd and should stop.

So... any further comment on this? Shall I make it a proper contribution or do we ditch the idea?
I'm in no rush, but it's better to avoid having open PR without actioning them.


FPS counter with AntTweakBar

This does not work on linux yet it seems.

I'm trying to make it work now. Installed anttweakbar manually


Removed floating $ in findBLOSC.cmake causing it to fail

Codecov Report

Merging #104 (69612ea) into master (b338ac9) will decrease coverage by 19.41%.
The diff coverage is n/a.

:exclamation: Current head 69612ea differs from pull request most recent head eb6d8b6. Consider uploading reports for the commit eb6d8b6 to get more accurate results

@@             Coverage Diff             @@
##           master     #104       +/-   ##
===========================================
- Coverage   62.45%   43.03%   -19.42%
===========================================
  Files          57       55        -2
  Lines       17686    15071     -2615
===========================================
- Hits        11045     6486     -4559
- Misses       6641     8585     +1944

Impacted Files
Coverage Δ

core/include/misc/utils.h
0.00% <0.00%> (-100.00%)
:arrow_down:

core/include/codec/codec_lz4.h
0.00% <0.00%> (-100.00%)
:arrow_down:

core/include/storage_manager/storage_buffer.h
0.00% <0.00%> (-100.00%)
:arrow_down:

core/src/misc/progress_bar.cc
0.00% <0.00%> (-96.78%)
:arrow_down:

core/src/codec/codec_lz4.cc
4.16% <0.00%> (-85.12%)
:arrow_down:

core/src/storage_manager/storage_buffer.cc
1.23% <0.00%> (-82.29%)
:arrow_down:

core/src/storage_manager/storage_s3.cc
0.65% <0.00%> (-79.87%)
:arrow_down:

core/src/storage_manager/storage_azure_blob.cc
0.36% <0.00%> (-76.89%)
:arrow_down:

core/include/storage_manager/storage_azure_blob.h
0.00% <0.00%> (-75.56%)
:arrow_down:

core/include/misc/uri.h
25.00% <0.00%> (-75.00%)
:arrow_down:

... and 40 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b338ac9...eb6d8b6. Read the comment docs.

@aoblebea you'll want to branch off of develop and PR to that.

@aoblebea, we don't need to do a find_package at all for Blosc and ZStd as they are dynamically loaded. I have made the changes in ng_codec07282021 branch. Will issue a PR against that soon. Thanks.


Make basis_transcoder.{js,wasm} substantially smaller

This change substantially reduces the size of Basis transcoder when built for Web; this means that the transcoder itself can download and compile faster.
The changes are as follows:

Rebuild transcoder using latest Emscripten (1.38.31)
Remove std::iostream (it has large impact on both JS and WASM code, and is only used for one console error message)
Make BASISD_SUPPORT macros configurable externally and disable BC7 for web build since BC7 doesn't seem to be currently supported in WebGL in practice (BC7 decoding tables are large)
Use emmalloc instead of default dlmalloc (allocations are slightly slower but there are very few of them so it doesn't impact transcoding time)

This change also cleans up the use of assert(x) by adding NDEBUG - this change isn't very significant on the code size, but removes a hack that's not necessary
The cumulative impact of the changes is as follows (individual impact is recorded in each commit message):
Before:
basis_transcoder.js 139216
basis_transcoder.wasm 886294
basis_transcoder.js.gz + basis_transcoder.wasm.gz 366383
After:
basis_transcoder.js 58707
basis_transcoder.wasm 217809
basis_transcoder.js.gz + basis_transcoder.wasm.gz 124984
There are some more gains to be had in wasm and JS code but they seem to come with performance tradeoffs or more complex build pipeline, so I've settled on this for now.
Re: BC7: it technically can be supported via EXT_texture_compression_bptc extension that's currently "community approved". However, neither Chrome nor Firefox implement it on Windows, and the compatibility table is uninspiring: https://developer.mozilla.org/en-US/docs/Web/API/EXT_texture_compression_bptc. Since BC7 tables take ~470 KB of wasm, I think it makes sense to compile this out for now, and add BC7 if and when it becomes a viable option in WebGL.

Thank you, this is awesome. Transcoder download size is very, very important so I will test and merge this in tomorrow (Thursday).
For BC7, I've been planning on reducing the size of the tables, so eventually when BC7 is supported more widely on the Web it can be enabled. BC7 doesn't add much extra value now, but it will once I get higher quality support in there.

It's in, thanks!


YOLO NAS'S Precision is significantly lower compare to other later YOLO model even when using same dataset ?

💡 Your Question
I was doing a object detection project , which is a drowning detection . Below is the result of YOLO-NAS S with 250 epochs.

Despite having high recall and mAP value , both precision and f1 score is low under (0.1) . When testing the model , it do quite well on detecting class despite having low precision .
So I try to find if other people face the same issue .
1.https://github.com/Deci-AI/super-gradients/issues/1191
2.https://github.com/Deci-AI/super-gradients/issues/1734
3.Performance Analysis of YOLO-NAS SOTA Models on CAL Tool Detection
YoloNASCALPaper-3.pdf

4.ASSESSING THE EFFECTIVENESS OF YOLO ARCHITECTURES FOR SMOKE AND WILDFIRE DETECTION
Assessing_the_Effectiveness_of_YOLO_Architectures_for_Smoke_and_Wildfire_Detection.pdf

5.COMPARATIVE STUDY OF YOLOV8 AND YOLO-NAS FOR AGRICULTURE APPLICATION
Comparative_study_of_YOLOv8_and_YOLO-NAS_for_agriculture_application.pdf

From the 5 cases above , it seem like having very low precision but at the same time high recall and mAP is quite normal for YOLO-NAS. When compare with other YOLO version with same dataset, the other YOLO model does not face the same problem . Why does this happen , considering that YOLO NAS generally perform better than other lower YOLO model? Is it because how different YOLO-NAS been evaluate ?
Versions
No response

Not an expert at all but how did you train your model ? I think (if you follow most tutorials), there is several data augmentations step performed by default (pretty hidden) on the training dataset.

@YangB-INSA , I using Roboflow to store and data augmentations , to train I use Google Colab notebook from here https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/train-yolo-nas-on-custom-dataset.ipynb#scrollTo=SdQ5JGblbJTk .

You can see the list of default data transforms done on your training data with train_data.dataset.transforms
If you want to remove some transforms that may or may not mess up your training you can remove them with train_data.dataset.transforms.pop(index).
You can plot a sample of your training data with the transforms with train_data.dataset.plot()
If you see the Mixup one , try to remove it and retrain your model.  Just test whatever you want from there.

It depends if you see the COCO dataset the average of the images have a pixel rate, if you are training to train imagen that the size are too different the results could change, so maybe if you resized your dataset would help


[dev] error when session goes stale

[13:03:21] Session stale, re-logging in
[13:03:21] Attempting login to Pokemon Go.
Traceback (most recent call last): File "/tools/PokemonGo-Bot/pokecli.py", line 283, in main() File "/tools/PokemonGo-Bot/pokecli.py", line 249, in main
bot.take_step()
File "/tools/PokemonGo-Bot/pokemongo_bot/init.py", line 47, in take_step self.process_cells(work_on_forts=True) File "/tools/PokemonGo-Bot/pokemongo_bot/init.py", line 67, in process_cells
"catchable_pokemons": catchable_pokemons}, location, work_on_forts) File "/tools/PokemonGo-Bot/pokemongo_bot/init.py", line 161, in work_on_cell
self.check_session(position)
File "/tools/PokemonGo-Bot/pokemongo_bot/init.py", line 227, in check_session
self.login()
File "/tools/PokemonGo-Bot/pokemongo_bot/init.py", line 240, in login
str(self.config.password)): File "/tools/PokemonGo-Bot/src/pgoapi/pgoapi/pgoapi.py", line 157, in login response = self.call()
File "/tools/PokemonGo-Bot/src/pgoapi/pgoapi/pgoapi.py", line 80, in call response = request.request(api_endpoint, self._req_method_list, player_position) File "/tools/PokemonGo-Bot/src/pgoapi/pgoapi/rpc_api.py", line 93, in request
response_dict = self._parse_main_response(response, subrequests) File "/tools/PokemonGo-Bot/src/pgoapi/pgoapi/rpc_api.py", line 201, in _parse_main_response
response_proto_dict = self._parse_sub_responses(response_proto, subrequests, response_proto_dict) File "/tools/PokemonGo-Bot/src/pgoapi/pgoapi/rpc_api.py", line 208, in _parse_sub_responses
del response_proto_dict['returns'] KeyError
: 'returns'

Looks like this has been fixed 5 days ago https://github.com/tejado/pgoapi/commit/1f25e907f3e5f1b603330e71041e1ad7bee7580f so we just need to bring the copy of pgoapi up to date.

same error here. any solutions yet?

pip.exe install  --upgrade git+https://github.com/tejado/pgoapi.git
fixed it for me.

Solution above. Closing this.
/closed


Install Keycloak

Need info for roles/permissions

@MightyNerdEric get installation done; @HeatherAck will set up meeting to define roles/groups/access permissions/etc.

@HeatherAck to set up mtg in afternoon. but not pre-req for install itself

@MightyNerdEric reviewing steps to install Keycloak to properly work with open shift cluster.  On pace to install this week (15-May)

Keycloak is installed on CL3, but this is just the initial step. I need to add a production-ready database container, and then write the config out to our ops-argocd repo. I'll be off the next couple days (June 2 & 5), but will resume work where I've left off as soon as I return.

For week of 12-Jun:  need to get production db connected to it, plus a few other tasks - need to be added to ArgoCD.

had issues with another project.  will prioritize this week.  need to try it out as operator (operator installed, needs to be deployed). @MightyNerdEric to update issue on Friday will status

I have not been able to spin up a stable Keycloak deployment using the operator. The other system has been ready for weeks, but I was tasked with comparing the two. So here is the result of my comparison: given the same configuration values, the manual setup (which will never be manual again, as we're going to put it into code) gave us a solid, stable system, and the operator did not. So we will move forward with the initial config. I will push the change for ops-argocd on Monday.

Quick update on this: the PR for the code change is open, but not quite complete. I'm still squishing a few bugs, but most of the components seem to be working as intended. I expect this to be done by EOW.

Keycloak not coming up due to postgres issue where it won't load.  Need to address postgres issue - maybe an init file issue, possibly not being run as root.  If cannot get to run easily, need to transition to running it as operator [also being used as data mesh pattern].  See example:  https://github.com/opendatahub-io-contrib/data-mesh-pattern/tree/main/gitops/keycloak/overlays
Most recent one:  https://github.com/keycloak/keycloak/tree/main/operator

pending code reviews - @redmikhail started review.  postgres not working.  will propose PR tomorrow morning to @MightyNerdEric branch. would like to use operator. installed in local env.  goal is to push to main week of 31-Jul

closed - but @redmikhail will create a new issue(s) to set up users, plus config and setup between data exchange and data commons


can filter out more projects in gitlab discovery

this is a very minor potential improvement to gitlab discovery, but i noticed when running gitlab discovery that backstage filters out "archived" projects but not "empty repo" projects (see: https://gitlab.com/gitlab-org/gitlab/-/issues/3800). these projects don't have a repository and thus don't have a default branch, though discovery assumes project.default_branch to exist and attempts to fetch a catalog file, erroring with:
catalog warn Unable to read url, Error: https://[gitlab-host]/[group]/[project]/-/blob/undefined/catalog-info.yaml could not be read as https://[gitlab-host]/api/v4/projects/[project-id]/repository/files/catalog-info.yaml/raw?ref=undefined, 403 Forbidden

not a huge deal but the warn logs are confusing: "why are we trying to discover from ?ref=undefined?" on a gitlab instance with a lot of empty repo projects, discovery could save some time by filtering these out up front.
it seems a decent way to do this is filter out project.empty_repo === true, though this isn't yet documented in their API. we could also filter where project.default_branch === undefined.
https://github.com/backstage/backstage/blob/5284458e18f7da3bc0ba5eb8043e902108518726/plugins/catalog-backend/src/ingestion/processors/GitLabDiscoveryProcessor.ts#L97-L99
just wanted to flag! happy to put up a PR if desired.

ah apologies, clicking "reference in new issue" on that code block didn't pull up the issue template. happy to re-format if folks prefer.

Thank you for reporting! I think the default_branch === undefined check makes sense if that's the only documented way to check it. No worries about the template

yep, that makes sense. if i have some time this week i may put up a PR. wouldn’t mind getting in the habit of it :)


unable to download playlist

Thanks for the amazing bot which helped me and my friends a lot.
easy to deploy and manage.
I was in the logs the bot by default use "--no-playlist".
I have a quick question can tell me where I can remove "--no-playlist" command to allow users to download playlists?
Thanks again.

No. Playlist support is not implemented.


Support custom manager URL across all commands

Most places that instantiate a new Harness API client are not currently passing the manager URL option, even though it is supported.  Update all places where the client is created to support this option

Need to test fix above with an actual on-prem manager.
There are also at least 10 other locations (mostly in the config as code/template commands) that need to be updated.


Update scalatest to 3.2.4

Updates org.scalatest:scalatest from 3.2.3 to 3.2.4.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scalatest", artifactId = "scalatest" } ]

labels: library-update, semver-patch

Codecov Report

Merging #927 (14b8771) into develop (3781b98) will increase coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #927      +/-   ##
===========================================
+ Coverage    89.61%   89.65%   +0.04%
===========================================
  Files          119      119
  Lines         9869     9869
  Branches       494      494
===========================================
+ Hits          8844     8848       +4
+ Misses        1025     1021       -4

Impacted Files
Coverage Δ

...witter/algebird/util/summer/AsyncListMMapSum.scala
96.15% <0.00%> (-3.85%)
:arrow_down:

.../main/scala/com/twitter/algebird/HyperLogLog.scala
89.61% <0.00%> (-0.44%)
:arrow_down:

.../scala/com/twitter/algebird/immutable/BitSet.scala
96.78% <0.00%> (-0.22%)
:arrow_down:

...src/main/scala/com/twitter/algebird/Interval.scala
81.73% <0.00%> (ø)

.../main/scala/com/twitter/algebird/BloomFilter.scala
93.36% <0.00%> (ø)

...a/com/twitter/algebird/immutable/BloomFilter.scala
95.90% <0.00%> (+1.63%)
:arrow_up:

.../main/scala/com/twitter/algebird/JavaMonoids.scala
79.66% <0.00%> (+1.69%)
:arrow_up:

...main/scala/com/twitter/algebird/monad/Reader.scala
58.82% <0.00%> (+5.88%)
:arrow_up:

...n/scala/com/twitter/algebird/SuccessibleLaws.scala
92.30% <0.00%> (+15.38%)
:arrow_up:

.../main/scala/com/twitter/algebird/RightFolded.scala
100.00% <0.00%> (+25.00%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3781b98...14b8771. Read the comment docs.


Allow setting tokens via env variables

I already have a lot of tokens in my ENV variables and would like to re-use them.

Some way to say "look at this envvar" from the config file would be a good way to do this. I guess making the password be something like:
token = "string" # current status
token = { value = "string" } # what it is interpreted as
token = { environment = "variable" }

This would also be possible:
token = { process = [ ... ] }

Guidance from https://github.com/rust-lang/rfcs/pull/2730 would be useful to know how to interpret the process case on different platforms.

I'm not fond of changing the configuration file just to get tokens from environment variables because it seems like a very narrow use case. It also encourages sharing the same token with multiple applications which is not ideal.
However the token-from-process used by cargo seems like a good idea since it should cover most use cases of people storing their secrets outside of the configuration file. I'm not too worried about cross platform compatibility since Golang provides exec.Command to run arbitrary commands and the signature of this function matches the value expected for token-from-process.
So setting a token could be done in a couple of ways:
# Token stored in the configuration file
token = "mytoken"
# Token stored in an environment variable
token-from-process = ["bash", "-c", "echo $MY_API_TOKEN"]
# Generic case: token returned by calling `executable arg1 arg2 ...`
token-from-process = ["executable", "arg1", "arg2"]

As mentioned in the RFC the process inherits stdin and stderr so people could store their tokens in their password manager and be prompted for a password when running cistern. It seems like a good solution. Thanks @mathstuf for pointing out the RFC.

It also encourages sharing the same token with multiple applications which is not ideal.

This is the exact use case I have. I don't want to generate 15 tokens for GitHub but have on per machine and share it with all my programs and using ENVs for something is not narrow. Basically any program which wants an API Token of some sort uses the ENV GITHUB_TOKEN.
Also I don't really see the need to store an API token which has the scope of showing publicly available information in a password manager.

Why would you need a token per project for cistern? Isn't the token per host in the configuration file?

I made the changes for this feature on the following branch: https://github.com/nbedos/cistern/tree/feature/token_from_process
Usage is detailed in the sample configuration file:
# Also, every "token" key can be replaced by "token-from-process" which is a list of strings
# consisting of the name of an executable to call followed by the arguments to be provided to it.
# The process will inherit stdin and stderr (allowing for password prompts) and the output of the
# process will be used as the value of the token.
#
# Examples:
#        # Read token from environment variable
#        token-from-process = ["bash", "-c", "echo $MY_API_TOKEN"]
#
#        # Read token from `pass` store
#        token-from-process = ["pass", "ci/gitlab/cistern"]
#

I tested this with pass and the workflow is quite nice. You get a single password prompt and then cistern starts as normal. It even works without an X server.
@SuperSandro2000  Let me know what you think.

Single password prompt even for multiple tokens?
I still kinda prefer getting environment variables directly.

Well you can do that by setting token-from-process = ["bash", "-c", "echo $MY_API_TOKEN"] and you won't be asked for a password.
pass is just a password store I tested but you don't have to use it all. But yes, it asks for a single password if all tokens are encrypted using the same GPG key.

This only works if I have the tokens in my startup script and is kinda a workaround.
Using os.Getenv("MY_API_TOKEN") would be preferable and just nicer.

This only works if I have the tokens in my startup script

This should be fixed by https://github.com/nbedos/cistern/commit/a4392aafb35b83399397a6ba90ba9c86cae86354

Using os.Getenv("MY_API_TOKEN") would be preferable and just nicer.

Yeah but then I have to implement, test and document both token-from-env and token-from-process when just the second configuration option covers both cases.
I understand the syntax is not ideal but it provides the functionality you requested and I don't see any other drawback. And given how it's implemented it will also work for other people storing their credentials elsewhere meaning it's unlikely I'll have to extend this feature later.

Yeah but then I have to implement, test and document both token-from-env and token-from-process when just the second configuration option covers both cases.

So right now we are both lazy and don't want to do something. I mean it should be fine. Thanks 👍

So right now we are both lazy and don't want to do something.

Just stop contributing to this repository. I'm sure you can find other repositories and developers that are up to your standards.


v2 webrtc e2e tests, relax expectation on packets received

Description
e2e v2 webrtc tests. Relax expectation in terms of packets received during a call.
Refactor that logic so that it can be reused.
Type of change

[x] Internal refactoring
[ ] Bug fix (bugfix - non-breaking)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Code snippets
In case of new feature or breaking changes, please include code snippets.

pps: packet per second.

Just a small suggestion

I don't understand the suggestion though @jpsantosbh - those are pps (packets per second). What am I missing?

NVM, You are dividing by 1000 (I missed that)


Fixes logic surrounding RequireFailure.

The order in which events were added to the Manager affected how RequireFailure works. This fixes issue #763

Here's an example of the bug: http://codepen.io/tomcheng/pen/rVdGGL

@tomcheng thank you we will be testing this out and hopefully merging soon


⚠ ClamAV has degraded performance

In 9f0a52e, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 94 ms

Resolved: ClamAV performance has improved in bf7f291.


Avoid array copy in md5 scalar

Looking at the underlying implementation the digest method doesn't use
the provided buffer directly but still does a copy:
protected int engineDigest(byte[] buf, int offset, int len)
                                                throws DigestException {
    byte[] digest = engineDigest();
    if (len < digest.length)
            throw new DigestException("partial digests not returned");
    if (buf.length - offset < digest.length)
            throw new DigestException("insufficient space in the output "
                                      + "buffer to store the digest");
    System.arraycopy(digest, 0, buf, offset, digest.length);
    return digest.length;
}

So we can use instead use the other overload and use the return value
directly to avoid the copy.

(The change is not measurable impact in a end-to-end benchmark as it's impact is tiny compared to all the remaining work happening during query processing)
Q: select md5('crate')
C: 20
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        3.555 ±   20.326 |      0.280 |      0.560 |      0.788 |    146.280 |
|   V2    |        3.279 ±   18.217 |      0.270 |      0.601 |      0.814 |    131.471 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               -   8.08%                           +   6.98%

retest this please
again:
test_graceful_stop_concurrent_queries (test_decommission.TestGracefulStopDuringQueryExecution) ... FAIL

======================================================================
FAIL: test_graceful_stop_concurrent_queries (test_decommission.TestGracefulStopDuringQueryExecution)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/CrateDB/crate_on_pr/blackbox/test_decommission.py", line 359, in test_graceful_stop_concurrent_queries
    self.assertEqual(errors, [])
AssertionError: Lists differ: [ProgrammingError("RelationUnknown[Relatio[73 chars]n]")] != []
First list contains 2 additional elements.
First extra element 0:
ProgrammingError("RelationUnknown[Relation 't1' unknown]")
+ []
- [ProgrammingError("RelationUnknown[Relation 't1' unknown]"),
-  ProgrammingError("RelationUnknown[Relation 't1' unknown]")]

Already having a look into that. It's unrelated to the change in this PR


Does FC domain BN introduce biases when inferencing?

Thanks for the excellent work and the highly-organized code.
I'm a little confused about the BN behavior of FC for CAN. According to what i observed, during the training, the target domain's FC BN was not updated. Only the source domain's FC BN layer was updated through the CE loss.
When testing, however, the target domain's FC BN was used. Would this harm the performance of the network? Since the target domain's FC BN haven't been trained and may introduce biases.
Thank you!

Maybe u can the the orignal papper:

all the network parameters are shared between the source domain and target domain data other than those of the batch normalization layers which are domain-specific

Maybe u can the the orignal papper:

all the network parameters are shared between the source domain and target domain data other than those of the batch normalization layers which are domain-specific

Yeah i read it. The domain-specific BNs make sense to me, as it can reduce different domain's feature discrepancy.
What i'm confused about is the behavior of last fully connected layer's BN1. when inferencing, the last fc's target domain's BN is used, which is never trained. I suspect it may introduce bias.
I would update my experiment on this issue later.

Hi, I don't think I have added a domain-specific BN layer to the last FC in my implementation. Could you point it to me if possible?

Maybe u can the the orignal papper:

all the network parameters are shared between the source domain and target domain data other than those of the batch normalization layers which are domain-specific

Yeah i read it. The domain-specific BNs make sense to me, as it can reduce different domain's feature discrepancy.
What i'm confused about is the behavior of last fully connected layer's BN1. when inferencing, the last fc's target domain's BN is used, which is never trained. I suspect it may introduce bias.
I would update my experiment on this issue later.

Perhaps there is a misunderstanding, the fact is that there are multiple BN layers in the network, each layer has BN0 and BN1. Use BN0 when training the source domain and BN1 when training the target domain.
                self.net.module.set_bn_domain(self.bn_domain_map[self.source_name])#switch to target source BN
                feats_source = self.net(source_cls_concat)
                self.net.module.set_bn_domain(self.bn_domain_map[self.target_name]) #switch to target domain BN
                feats_target = self.net(target_cls_concat)

                # prepare the features
                feats_toalign_S = self.prepare_feats(feats_source)
                feats_toalign_T = self.prepare_feats(feats_target)

                cdd_loss = self.cdd.forward(feats_toalign_S, feats_toalign_T,
                               source_nums_cls, target_nums_cls)[self.discrepancy_key]

                cdd_loss *= self.opt.CDD.LOSS_WEIGHT
                cdd_loss.backward()

Hi, I don't think I have added a domain-specific BN layer to the last FC in my implementation. Could you point it to me if possible?

Thanks for your reply. When defining FC of CAN model, it uses domain-specific BN (in class DANet)
self.FC[str(k)] = FC_BN_ReLU_Domain(in_dim, out_dim, num_domains_bn)

In update_network of CAN_solver.py, i found that when calculating ce_loss, it uses source domain's BN, i.e. the source domain's FC's BN was updated.
self.net.module.set_bn_domain(self.bn_domain_map[self.source_name])
source_preds = self.net(source_data)['logits']

# compute the cross-entropy loss
ce_loss = self.CELoss(source_preds, source_gt)
ce_loss.backward()

When testing, however, the net's bn domain was set to be target domain (TEST.DOMAIN), i.e. the FC's target domain's BN was used, which haven't been trained. (in test.py)
if cfg.TEST.DOMAIN in bn_domain_map:
    domain_id = bn_domain_map[cfg.TEST.DOMAIN]
else:
    domain_id = 0

with torch.no_grad():
    net.module.set_bn_domain(domain_id)
    for sample in iter(dataloader):
        res['path'] += sample['Path']

        if cfg.DATA_TRANSFORM.WITH_FIVE_CROP:
            n, ncrop, c, h, w = sample['Img'].size()
            sample['Img'] = sample['Img'].view(-1, c, h, w)
            img = to_cuda(sample['Img'])
            probs = net(img)['probs']
            probs = probs.view(n, ncrop, -1).mean(dim=1)
        else:
            img = to_cuda(sample['Img'])
            probs = net(img)['probs']

Maybe u can the the orignal papper:

all the network parameters are shared between the source domain and target domain data other than those of the batch normalization layers which are domain-specific

Yeah i read it. The domain-specific BNs make sense to me, as it can reduce different domain's feature discrepancy.
What i'm confused about is the behavior of last fully connected layer's BN1. when inferencing, the last fc's target domain's BN is used, which is never trained. I suspect it may introduce bias.
I would update my experiment on this issue later.

Perhaps there is a misunderstanding, the fact is that there are multiple BN layers in the network, each layer has BN0 and BN1. Use BN0 when training the source domain and BN1 when training the target domain.
                self.net.module.set_bn_domain(self.bn_domain_map[self.source_name])#switch to target source BN
                feats_source = self.net(source_cls_concat)
                self.net.module.set_bn_domain(self.bn_domain_map[self.target_name]) #switch to target domain BN
                feats_target = self.net(target_cls_concat)

                # prepare the features
                feats_toalign_S = self.prepare_feats(feats_source)
                feats_toalign_T = self.prepare_feats(feats_target)

                cdd_loss = self.cdd.forward(feats_toalign_S, feats_toalign_T,
                               source_nums_cls, target_nums_cls)[self.discrepancy_key]

                cdd_loss *= self.opt.CDD.LOSS_WEIGHT
                cdd_loss.backward()

Thank you for your reply.
I agree with what you said. There're relative BNs for source & target domain, while both domain shares the weight of conv layer. My problem is that, the last fully connected layers (not feature extractor) also contain domain-specific BN (if i didn't mis-interpret the code). The target domain's BN is not updated during training while it is used for inferencing.

@Ledzy I think you misunderstood this part. First, both the source domain BN and the target domain BN are updated during training. The CE loss is imposed on the source data, thus you should switch the domain BN to the source mode. While for the alignment with CDD, I just switch the domain BN to the target mode before I obtain the target feature. You may refer to the  Line 142 and Line 253 in the solver/can_solver.py. Second, I just use one layer FC. Thus in my provided configuration, actually the network don't have any FC BN layer. But you can choose to use mutlilayer-FC, which has the FC domain BN layer in this case. And it will work correctly as I just mentioned. Hope this may help you.


pip install cdl error

pip install cdl in fresh python3 environment on MacOS 12.6.9 fails on install of PlotPy with error:
Building wheels for collected packages: PlotPy
  Building wheel for PlotPy (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for PlotPy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      ./src/scaler.cpp:58:35: error: a space is required between consecutive right angle brackets (use '> >')
      typedef XYTransform<Array1D<double>> XYScale;
                                        ^~
                                        > >
      ./src/scaler.cpp:616:56: error: a space is required between consecutive right angle brackets (use '> >')
          return scale_src_dst<Params, NoScale<ST, npy_uint32>>(p, scale);
                                                             ^~
                                                             > >
      2 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for PlotPy
Failed to build PlotPy
ERROR: Could not build wheels for PlotPy, which is required to install pyproject.toml-based projects

Is this not the correct way to install DataLab on MacOS?

Thanks for the feedback and for trying to play with DataLab!
Would you be able to try something to help me understand what's going on?
This would require to clone the PlotPy repository and to slightly change the setup.py script by adding the following line just before line 55 (before the call to the setup function):
CFLAGS_CPP += ["-std=c++11"]

And then try and build PlotPy from source:
python setup.py build

just to see if it works.

This seems to have worked. At least I get no errors during the build.

Great, thanks for the feedback.
I'll try to release PlotPy v2.0.3 as quickly as possible with this change.
In the meantime, if you type:
python setup.py build install

...you may retry installing DataLab with pip install cdl.

Thanks for the quick fix @PierreRaybaut !!!

I was able to run cdl from command line and I get the UI window. Not sure if it really matters but FYI, I also get a lot of warnings stating ModuleNotFoundError: No module named 'plotpy._scaler'

Those warnings mean that you haven't build the  C++ extensions, or at least that the plotpy module that is imported is missing those extensions. Maybe have you run the cdl command without having change the current directory from your previous build commands and you are importing the plotpy module from the repository instead of the one that you have just installed?

Sorry if this should really be asked elsewhere, but I can't figure out based on the docs how to do much other than draw an ROI and see its basic stats. For example, with a signal selected (I just created a sinusoid via the UI) I only have Edit, View, and ? menus, but no menus for Operation, Processing, Computing? How might I fit a function to a curve? Or is this not yet possible?

You nailed it! The module not found error is gone if I run cdl from another directory!
But now I only have Edit and ? menus with a signal selected.

You nailed it! The module not found error is gone if I run cdl from another directory!
But now I only have Edit and ? menus with a signal selected.

Could please drag and drop a screenshot?
The fact that some of the menus are not shown is clearly not normal. I'm not sure why it's happening.

You nailed it! The module not found error is gone if I run cdl from another directory!
But now I only have Edit and ? menus with a signal selected.

Could please drag and drop a screenshot?
The fact that some of the menus are not shown is clearly not normal. I'm not sure why it's happening.

Are there any traceback in the console?

You nailed it! The module not found error is gone if I run cdl from another directory!
But now I only have Edit and ? menus with a signal selected.

Could please drag and drop a screenshot?
The fact that some of the menus are not shown is clearly not normal. I'm not sure why it's happening.

Any traceback in the console?

Also, you may try and add a "DEBUG" environment variable and set it to "1", for a time being: this will redirect stdin/stout to the terminal (instead of the embedded Qt comsole).

Only have Edit and ? menus with a signal selected.

Only have Edit and ? menus with a signal selected.

Thanks a lot for the screenshot.
That's really unexpected. I assume it's specific to the MacOS platform as we've thoroughly tested DataLab on both Windows and Linux (at least CentOS and Ubuntu).
I'm still interested in any traceback or warning you may have, with or without the DEBUG environment variable.
Plus, if you could share your Python install details, it surely could help understanding what's going on (e.g. the output of pip list).

I have found some interesting leads on how to solve this missing menu issue (and I confirm that it seems to be a MacOS specific issue).
Some things could be done easily, and if I can come up with something, I'll add a fix with the next release (scheduled for tomorrow if everything's ok).

Odd deprecation warning in output of pip list:
DEPRECATION: Loading egg at XXX/miniconda3/envs/datalab/lib/python3.12/site-packages/PlotPy-2.0.2-py3.12-macosx-10.9-x86_64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
DEPRECATION: Loading egg at XXX/miniconda3/envs/datalab/lib/python3.12/site-packages/tifffile-2023.12.9-py3.12.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
Package                       Version
----------------------------- ----------
aiohttp                       3.9.1
aiosignal                     1.3.1
alabaster                     0.7.13
astroid                       3.0.2
attrs                         23.1.0
Babel                         2.14.0
black                         23.12.0
cdl                           0.9.2
certifi                       2023.11.17
charset-normalizer            3.3.2
click                         8.1.7
coverage                      7.3.4
Cython                        3.0.7
dill                          0.3.7
docutils                      0.20.1
frozenlist                    1.4.1
guidata                       3.2.2
h5py                          3.10.0
idna                          3.6
imageio                       2.33.1
imagesize                     1.4.1
iniconfig                     2.0.0
isort                         5.13.2
Jinja2                        3.1.2
lazy_loader                   0.3
markdown-it-py                3.0.0
MarkupSafe                    2.1.3
mccabe                        0.7.0
mdit-py-plugins               0.4.0
mdurl                         0.1.2
multidict                     6.0.4
mypy-extensions               1.0.0
myst-parser                   2.0.0
networkx                      3.2.1
numpy                         1.26.2
opencv-python                 <IP_ADDRESS>
packaging                     23.2
pathspec                      0.12.1
Pillow                        10.1.0
pip                           23.3.2
platformdirs                  4.1.0
PlotPy                        2.0.2
PlotPy                        2.0.2
pluggy                        1.3.0
psutil                        5.9.7
Pygments                      2.17.2
pylint                        3.0.3
PyQt5                         5.15.10
PyQt5-Qt5                     5.15.11
PyQt5-sip                     12.13.0
pytest                        7.4.3
pytest-cov                    4.1.0
pytest-qt                     4.2.0
pytest-xvfb                   3.0.0
python-docs-theme             2023.9
PythonQwt                     0.11.2
PyVirtualDisplay              3.0
PyWavelets                    1.5.0
PyYAML                        6.0.1
QtPy                          2.4.1
requests                      2.31.0
scikit-image                  0.22.0
scipy                         1.11.4
setuptools                    68.2.2
snowballstemmer               2.2.0
Sphinx                        7.2.6
sphinx-copybutton             0.5.2
sphinx-qt-documentation       0.4.1
sphinxcontrib-applehelp       1.0.7
sphinxcontrib-devhelp         1.0.5
sphinxcontrib-htmlhelp        2.0.4
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.6
sphinxcontrib-serializinghtml 1.1.9
tifffile                      2023.12.9
tifffile                      2023.12.9
tomli                         2.0.1
tomlkit                       0.12.3
urllib3                       2.1.0
wheel                         0.42.0
yarl                          1.9.4

Console output when running cdl with env DEBUG=1:
*** DEBUG mode *** [Reset configuration file, do not redirect std I/O]
<cdl.core.gui.main.CDLMainWindow object at 0x1374e7410>: Starting initialization
<cdl.core.gui.main.CDLMainWindow object at 0x1374e7410>: 2 plugin(s) found
<class 'cdl.plugins.PluginRegistry'>: Plugin Test data registered
qt.qpa.fonts: Populating font family aliases took 227 ms. Replace uses of missing font family "Default" with one that exists to avoid this cost.
*** DEBUG mode *** [Reset configuration file, do not redirect std I/O]
<cdl.core.gui.main.CDLMainWindow object at 0x1374e7410>: Initialization done
*** DEBUG mode *** [Reset configuration file, do not redirect std I/O]

Thanks @PierreRaybaut for looking into this so quickly. If you find a simple fix for MacOS that would be great!

Part 1. of the issue is now fixed with PlotPy v2.0.3.

So, this should now be fixed in DataLab v0.10.
I will release this new version in a few minutes/hours.

Part 2. of the issue has been fixed by 384296f6d6ca50ba1a8a08aab56eb23ae63962fa.
DataLab v0.10.0 includes this fix.
I'll let the issue opened until you can confirm this is really fixed on your side (I was not able to test the fix on MacOS).

Wow, thanks for the incredibly fast fix!
I confirm that now pip install PyQt5 cdl; cdl launches UI with all menus available.
P.S. I see options for a few fits such as Gaussian, etc. But is there a generic curve fitting UI, e.g. for a custom equation? Would be nice to have a spline fit as well. Or do I need to just implement those via a plugin?

Wow, thanks for the incredibly fast fix! I confirm that now pip install PyQt5 cdl; cdl launches UI with all menus available on MacOS 12.6.9.

Great news, thanks for testing!

P.S. I see options for a few fits such as Gaussian, etc. But is there a generic curve fitting UI, e.g. for a custom equation? Would be nice to have a spline fit as well. Or do I need to just implement those via a plugin?

Currently, there is no generic curve fitting UI but that may be implemented in the near future: please do not hesitate to create a "Feature request" here.
Same for the spline fit: please do not hesitate to create a "Feature request" here.
Or, you may also implement those via a plugin, indeed.
Even better, you may contribute for this feature to be available in the main application.


feat: Make openFile experience consistent with built-in trees

This PR changes two parts:

From VS Code 1.52, all custom tree behavior has been changed, which enable our plugin to have more consistent experience with file explorer when open file. That means, all following interactions are enabled for fileNode and primaryTypeNode:

Double-click or mouse-middle-click to open in non-preview.
Enter to open in non-preview mode (Space to open in preview mode).
Alt+Click to open an editor to the side of the active one.

reference: https://code.visualstudio.com/updates/v1_52

Since changes above needs to register vscode.open as TreeItem's command id, we can not use customized java.view.package.openFile command to send telemetry. As compensate, we send these additional info in telemetry:

{projectManagerVisible: 1} when project manager is expanded
counter for didChangeSelection, didCollapseElement, didExpandElement events

Thanks @xqzlgy2


UI Overhaul

Codecov Report

Merging #92 into master will decrease coverage by 2.53%.
The diff coverage is 7.36%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa70c9d...dfce6b3. Read the comment docs.

Closes #93

The refined UI looks elegant!


87 mi wo ex

added required files into the correct package (ConcreteOntology)

Why are there so many files here (https://github.com/BAMresearch/LebeDigital/tree/87-MiWoEx/lebedigital/ConcreteOntology)?
It looks to me that there are some old versions still in this folder.

@soudehMasoudian  could you check for the merge conflicts and then merge

@joergfunger the conflict resolved and merged.


Bump outlines version to 0.1.11 to include ARM support, FIX #11178

Bump outlines version to 0.1.11. Currently vLLM depends on outlines v0.1.9 which was breaking on ARM due to missing arch support on outlines-core for ARM (https://github.com/dottxt-ai/outlines-core/issues/122).
outlines has resolved this issue in their latest release v0.1.11.
FIXES #11178

#11140 should go in soon, which includes the version bump for outlines

Closing as superseded by #11140


Readd variable_type to select panel

I think this is all that is needed. I am able to select and retrieve the derived vars. I did notice that ck.view returns a blank map for these data though. I am not sure if this was an issue before or if I have missed something?

Indices now showing up for Fosburg and NOAA heat index! No problems retrieving and plotting Fosburg, but can't seem to retrieve the heat index data
NameError                                 Traceback (most recent call last)
Cell In [8], line 1
----> 1 data_to_use = selections.retrieve()

File ~/src/climakitae/climakitae/core/data_interface.py:1177, in DataParameters.retrieve(self, config, merge)
   1173     else:
   1174         raise ValueError(
   1175             "To retrieve data specified in a configuration file, please input the path to your local configuration csv as a string"
   1176         )
-> 1177 return read_catalog_from_select(self)

File ~/src/climakitae/climakitae/core/data_load.py:796, in read_catalog_from_select(selections)
    794     da = _get_fosberg_fire_index(selections)
    795 elif orig_var_id_selection == "noaa_heat_index_derived":  # Hourly
--> 796     da = _get_noaa_heat_index(selections)
    797 elif orig_var_id_selection == "effective_temp_index_derived":
    798     da = _get_eff_temp(selections)

File ~/src/climakitae/climakitae/core/data_load.py:1295, in _get_noaa_heat_index(selections)
   1293 selections.variable_id = ["t2"]
   1294 selections.units = "K"  # Kelvin required for humidity and dew point computation
-> 1295 t2_da_K = _get_data_one_var(selections, cat)
   1297 # Derive relative humidity
   1298 # Returned in units of [0-100]
   1299 rh_da = compute_relative_humidity(
   1300     pressure=pressure_da,  # Pa
   1301     temperature=t2_da_K,  # Kelvin
   1302     mixing_ratio=q2_da,  # kg/kg
   1303 )

NameError: name 'cat' is not defined

Thanks for

Indices now showing up for Fosburg and NOAA heat index! No problems retrieving and plotting Fosburg, but can't seem to retrieve the heat index data
NameError                                 Traceback (most recent call last)
Cell In [8], line 1
----> 1 data_to_use = selections.retrieve()

File ~/src/climakitae/climakitae/core/data_interface.py:1177, in DataParameters.retrieve(self, config, merge)
   1173     else:
   1174         raise ValueError(
   1175             "To retrieve data specified in a configuration file, please input the path to your local configuration csv as a string"
   1176         )
-> 1177 return read_catalog_from_select(self)

File ~/src/climakitae/climakitae/core/data_load.py:796, in read_catalog_from_select(selections)
    794     da = _get_fosberg_fire_index(selections)
    795 elif orig_var_id_selection == "noaa_heat_index_derived":  # Hourly
--> 796     da = _get_noaa_heat_index(selections)
    797 elif orig_var_id_selection == "effective_temp_index_derived":
    798     da = _get_eff_temp(selections)

File ~/src/climakitae/climakitae/core/data_load.py:1295, in _get_noaa_heat_index(selections)
   1293 selections.variable_id = ["t2"]
   1294 selections.units = "K"  # Kelvin required for humidity and dew point computation
-> 1295 t2_da_K = _get_data_one_var(selections, cat)
   1297 # Derive relative humidity
   1298 # Returned in units of [0-100]
   1299 rh_da = compute_relative_humidity(
   1300     pressure=pressure_da,  # Pa
   1301     temperature=t2_da_K,  # Kelvin
   1302     mixing_ratio=q2_da,  # kg/kg
   1303 )

NameError: name 'cat' is not defined

Thanks for catching that! I have fixed that issue.


Fix excessive checkStateNotNull in JdbcUtil

Fixes #25819

Also fix cannot write null values in derby (unit test)

The following error encountered with the new test:

java.lang.RuntimeException: java.lang.RuntimeException: Error while setting data to preparedStatement
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException:
...
Caused by: java.sql.SQLFeatureNotSupportedException: The data type '0' is not supported.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.checkForSupportedDataType(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedPreparedStatement.setNull(Unknown Source)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.setNull(DelegatingPreparedStatement.java:499)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.setNull(DelegatingPreparedStatement.java:499)
	at org.apache.beam.sdk.io.jdbc.JdbcUtil.setNullToPreparedStatement(JdbcUtil.java:272)
	at org.apache.beam.sdk.io.jdbc.JdbcUtil.lambda$static$aa0b0c57$3(JdbcUtil.java:109)
	at org.apache.beam.sdk.io.jdbc.JdbcIO$WriteVoid$AutoGeneratedPreparedStatementSetter.lambda$setParameters$1(JdbcIO.java:2207)
	... 19 more
Caused by: ERROR 0A000: The data type '0' is not supported.
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
	... 30 more

The data type '0' is NULL. Currently NULL primitive values are set by setNullToPreparedStatement(ps, i + 1, JDBCType.NULL);. It appears that this is working for some drivers (e.g. postsql), but not other (e.g. derby). However, the correct way should be to specify the SQL Type anyway.
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Thanks those are GItHub Action check flakes unrelated @aromanenko-dev


test(quick-install): Enable quick and experimential installation in CI

Fixes: #10616
Enable quick and experimential installation in CI

@aanm yes, i saw a few issues recently with ipv6, so just thought about adding related test as well. Will do it tomorrow in separate PR if you don’t mind.

@aanm yes, i saw a few issues recently with ipv6, so just thought about adding related test as well. Will do it tomorrow in separate PR if you don’t mind.

great! thank you!

Coverage increased (+0.008%) to 37.203% when pulling 93c81d02a8ea70622ddd8d9c6136882373769df9 on sayboras:feature/quick-install into c1f917ce1593113ce631333bb1a23196bdc20c98 on cilium:master.

@aanm @sayboras I've been trying to have an IPv6 only cluster running with kind lately and ran into multiple issues (probably some that you saw). Some of the required fixes are already merged into master but it's not sufficient yet to have an IPv6 only cluster working with kind. You'll likely run into more issues when working on this @sayboras so feel free to ping me if you need help.

I don't think GitHub Actions support IPv6 anyway: https://github.community/t/github-actions-ipv6-network-issues/18004.

I don't think GitHub Actions support IPv6 anyway: https://github.community/t/github-actions-ipv6-network-issues/18004.

Oh right, I am using local machine as self-hosted github action for debugging purpose before sending PR, and my ubuntu is having IPv6 enabled. Maybe that is why I have impression that github action supports ipv6 :cry:

hi, just want to highlight post-merge action for this PR. As I have done some renaming on github action job, we might need to change required check in github accordingly. Currently, all the builds will just have quick-install status waiting. Thanks.

hi, just want to highlight post-merge action for this PR. As I have done some renaming on github action job, we might need to change required check in github accordingly. Currently, all the builds will just have quick-install status waiting. Thanks.

/cc @aanm


Use a Map to track adverts on the page

What does this change?

Remove the advertIds property from dfpEnv and instead store adverts as a Map from their string ids to the adverts themselves. This allows us to store the same information in a single structure, with clearer semantics around getting/setting elements.

Remove the unused creativeIds property. It appears this array is only ever pushed to, and the values never retrieved from it.

Why?
Should be clearer when managing which adverts are being tracked in the dfpEnv.
In the browser console, for window.guardian.commercial.dfpEnv.adverts:

Nice refactor! Following #1121 we'll just have to update this line here too:
https://github.com/guardian/commercial/blob/c1dd4641048c86b96b054f00fba52e8e3d86c0a4/src/lib/header-bidding/prebid/bid-config.ts#L271

Thanks for letting me know!