Document the log message of webhook pod, when using the yaml generated by the serving commit 143c72e47c3076e9d392c7c90c60f33da649554a

I ran with a local kube on docker-desktop.
Try to install the operator of serving, with yaml generated by this commit https://github.com/knative/serving/commit/143c72e47c3076e9d392c7c90c60f33da649554a:
The webhook pod is unable to launch. The log of webhook pod is as below:

2019/11/04 17:40:24 Registering 2 clients
2019/11/04 17:40:24 Registering 0 informer factories
2019/11/04 17:40:24 Registering 0 informers
{"level":"info","ts":"2019-11-04T17:40:24.662Z","caller":"logging/config.go:108","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-11-04T17:40:24.662Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2019-11-04T17:40:24.733Z","logger":"webhook","caller":"profiling/server.go:60","msg":"Profiling enabled: false","commit":"33e4197","knative.dev/controller":"webhook"}
{"level":"info","ts":"2019-11-04T17:40:24.738Z","logger":"webhook.config-store","caller":"configmap/store.go:157","msg":"defaults config \"config-defaults\" config was added or updated: &config.Defaults{RevisionTimeoutSeconds:300, MaxRevisionTimeoutSeconds:600, UserContainerNameTemplate:\"user-container\", ContainerConcurrency:0, RevisionCPURequest:(*resource.Quantity)(nil), RevisionCPULimit:(*resource.Quantity)(nil), RevisionMemoryRequest:(*resource.Quantity)(nil), RevisionMemoryLimit:(*resource.Quantity)(nil)}","commit":"33e4197","knative.dev/controller":"webhook"}
{"level":"info","ts":"2019-11-04T17:40:24.738Z","logger":"webhook","caller":"metrics/exporter.go:97","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"33e4197","knative.dev/controller":"webhook"}
{"level":"info","ts":"2019-11-04T17:40:24.738Z","logger":"webhook","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/serving webhook prometheus<PHONE_NUMBER> 9090  false  false  }. Start the server for Prometheus exporter.","commit":"33e4197","knative.dev/controller":"webhook"}
{"level":"info","ts":"2019-11-04T17:40:24.738Z","logger":"webhook","caller":"metrics/exporter.go:106","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/serving webhook prometheus<PHONE_NUMBER> 9090  false  false  }","commit":"33e4197","knative.dev/controller":"webhook"}
{"level":"error","ts":"2019-11-04T17:40:24.842Z","logger":"webhook","caller":"webhook/webhook.go:160","msg":"could not configure admission webhook certs","commit":"33e4197","knative.dev/controller":"webhook","error":"server key missing","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/webhook.(*Webhook).Run\n\t/Users/vincent/go/src/knative.dev/serving/vendor/knative.dev/pkg/webhook/webhook.go:160\nmain.main.func2\n\t/Users/vincent/go/src/knative.dev/serving/cmd/webhook/main.go:198\nknative.dev/serving/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1\n\t/Users/vincent/go/src/knative.dev/serving/vendor/golang.org/x/sync/errgroup/errgroup.go:58"}
{"level":"error","ts":"2019-11-04T17:40:24.843Z","logger":"webhook","caller":"webhook/main.go:209","msg":"Error while running server","commit":"33e4197","knative.dev/controller":"webhook","error":"server key missing","stacktrace":"main.main\n\t/Users/vincent/go/src/knative.dev/serving/cmd/webhook/main.go:209\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

The newly generated yaml has added the following section:
apiVersion: v1
kind: Secret
metadata:
  labels:
    serving.knative.dev/release: devel
  name: webhook-certs
  namespace: knative-serving

@evankanderson is working on a PR for knative/pkg https://github.com/knative/pkg/pull/843, which targets to fix this issue.

@evankanderson has fixed it with a patch in manifestival package: https://github.com/jcrossley3/manifestival/commit/acb011d8dce378c23eadd735838a0aafd9b172b2


Implement binding types for var/let/const variables.

Work in progress! This doesn't pass the full test suite yet, and I'm not sure why yet. This aims to fix #5138 - it makes the register() method in the JavaScript highlighting mode aware of the type of binding it is in, and also adds block context that contains block-scoped definitions. Review + hints for how to get to the finish line much appreciated!

Nice. I'd keep a single chain of scopes, rather than separate ones for function and block scopes, though. You can add a property that describes the type of the scope.

Okay, refactored to use a single chain of prototypes. Still pretty baffled by what's going wrong with the indentation failures, though - I can see that it's now setting align=true when re-entering the mode, and that's aligning certain blocks incorrectly, but how that connects to the very minimal code change is a mystery.

It works! May also want to implement for scopes.

Merged as 40027e21423 and adjusted in 2e99e1414ac to avoid allocating too many objects


Fetch charts with GET to benefit from browser cache and conditional requests

This is a small PR that does a lot. It changes the initial request for charts (in explore or dashboards) to be done through a GET request, greatly improving the loading speed of dashboards. It also moves the caching to the HTTP layer, allowing us to benefit from Expires and ETag headers for conditional requests.
The problem
This diagram compares the current flow ("before") with the one implemented by this PR ("after"):

Before
Let's assume Superset is configured with a 1 hour cache, and also that the data changes on a longer period (daily, eg):

User "A" requests a chart from Superset doing a POST request with the payload.
Superset computes the query and sends it to the DB.
DB returns a dataframe.
Superset caches the dataframe.
Superset serializes the payload and sends it back to user "A".
User "A" refreshes the dashboard.
Superset finds the dataframe cached.
Superset serializes the payload and send it back to user "A".
Superset cache expires after 1 hour.
User "A" refreshes the dashboard.
Superset computes the query and sends it to the DB.
DB returns the exact same dataframe.
Superset caches the dataframe again.
Superset serializes the payload and sends it back to user "A".

There are a few inefficiencies here:

The browser cache is never used, because it's doing POST requests.
Superset needs to serialize the payload even on a cache hit.
Data is transferred to the browser even if it hasn't changed.

After

User "A" requests a chart from Superset doing a GET request with the chart id.
Superset computes the query and sends it to the DB.
DB returns a dataframe.
Superset serializes the dataframe and caches the HTTP response.
Superset sends the payload to user "A", with an Expires header of 1 hour, and an ETag header which is a hash of the payload.
The browser stores the response in its native cache, and SupersetClient caches it also in the Cache interface.
The user refreshes the dashboard.
Because of the Expires header and the use of GET the data is read directly from the native browser cache.
Superset cache expires after 1 hour.
User "A" refreshes the dashboard. The native cache is not used, since Expires is now in the past. SupersetClient looks for a cached response in the Cache interface, and if one is found, extracts its ETag.
The browser requests the chart with an If-None-Match header, containing the hash of the cached response (its ETag).
Superset computes the query and sends it to the DB.
DB returns the exact same dataframe.
Superset serializes the dataframe and caches the HTTP response.
Superset sees that the ETag matches the If-None-Match header, returning a 304 Not Modified response.
Browser fetches the cached response from the Cache interface.
Browser uses the response.

Notes

The GET request is done only the first time the chart is mounted. Forcing refresh on dashboards and clicking "Run Query" in the Explore views perform POST requests, which bypass the cache, and cache the new response. I tested the Explore view and dashboards with filters, and all further interactions are done with POSTs.

Since we're caching the HTTP response, we need to verify that the user has permission to read the cached response. This is done by passing a check_perms function to the decorator that caches the responses.

The fetch API has no support for conditional responses with ETags. We need to add explicit support in SupersetClient. I have a separate PR for that.

There is one small downside to this approach. During the time while Expires is still valid, the browser will not perform any requests for cached charts unless the user explicitly refreshes a dashboard or click "Run Query" in the Explore view. If the data is bad, they will see bad data until it expires or they purposefully refresh the chart. In the current workflow, in theory we can purge the cache in this case, since it lives only on the server-side. This is a hypothetical scenario, and we could workaround it by sending a notification to dashboards that one or more charts have bad data and should be refreshed.

:eyes @diggidydave

you sure GET request can handle this.props.formData? in airbnb many, many chart's formData are longer than 4k chars :)

@graceguo-supercat:

you sure GET request can handle this.props.formData? in airbnb many, many chart's formData are longer than 4k chars :)

Thanks, I'm aware of that problem. Here the GET request has only the chart ID, and the form data is read from the saved chart (params in the slices table is used).

why not use Redis cache query results, in aribnb the round trip to fetch data from Redis is only 600ms~800ms.

The decorator is using Redis for the server-side caching, but it's caching the HTTP response instead of the dataframe (saving time that's spent in serialization). But I'm also using the native browser cache (through the Expires header) and the new "Cache interface API" (in a separate PR that I'm finishing).

i think we should use etag for dashboard metadata (like dashboard layout, huge json blob...)

Everywhere! :)

Here the GET request has only the chart ID, and the form data is read from the saved chart (params in the slices table is used).
how about in dashboard with filter, each chart's query is not the saved params.
Also, you still have a formData in request parameters right? this is the blocker that cause issue.

how about in dashboard with filter, each chart's query is not the saved params? the actual query is formData overwrite saved chart's params.

Ah, you're right. The filter box works when changed, since it does a POST. But in the initial load it's not taken into consideration. Let me see how I can fix that.

Also, you still have a formData in GET request parameters right (you can see request parameters from browser location bar)? this is the blocker that cause issue.

Yes, but it has only the slice_id in it right now, eg: form_data: {"slice_id":78}. I'll see if I can append any additional parameters that are set by filters, this should keep it small.

@graceguo-supercat I tested the interaction with filter boxes and it's not working. I'll work on fixing it.

@betodealmeida this seems pretty complicated on top of data requests that are already complicated
and error prone 🙉
My first question to gauge whether it's worthwhile is: what are the speedup times you are seeing for existing approach vs your new approach? (ideally for multiple dashboards of varying size)

@williaster the speedup will greatly depend on how often the data changes, how big the payload is (bignum vs deck.gl varies significantly), the duration of the cache, and how slow the network is. I can (and have) run tests against the example dashboards, but I don't think they would be significative, since they don't cover all the real life use cases.
I think the question we should ask here is: "given that this is clearly an improvement, how can we make it bug free?"

@betodealmeida just to clarify step 4 (and per your diagram) you have:

Before: Superset caches the dataframe.
After: Superset serializes the dataframe and caches the HTTP response.

yet in the code it still seems like we're caching the result set from the database and thus I wonder if the diagram and thus After phase should mention that there's an additional entry in the server cache, i.e., after step (4) the server-side cache would contain:

The cached result set (as a dataframe).
The cached superset/explore_json HTTP response which includes server-side Python visualization specific mutations.

Note I'm not saying this is wrong as I strongly believe that the database response should be cache given that represents the bulk of the compute, I just wanted to get clarity on the logic.

@betodealmeida this is a big change that has the potential to impact many many users. If you're unable to provide numbers that indicate that this is strictly an improvement (or at a minimum no regressions), I'm a bit reluctant to introduce this additional complexity.
I think it's part of the expected work of a feature like this to demonstrate the effects with real-life examples. It seems like you should be able to use real Lyft dashboards, dashboards from the "example datasets", etc., and you can throttle network speed using dev tools.
Another concern I have is the impact on the # of requests. We've needed to introduce domain sharding because of the large number of simultaneous requests made by larger dashboards, and this potential DOUBLES that number, so I would want to see perf numbers that demonstrate no regressions for that case as well.

Not directly related, but on the topic of optimization around caching, it would be an extra win to make the caching call async. Maybe a first use case for the async/await syntax in Superset.

@mistercrunch fetch (SupersetClient) is async? do you just mean the async/await syntax?

@williaster I'm thinking about something else: what I mean on the server side, in python, making the cache.set(...) call async, meaning while a thread is pushing to the caching backend, the web server can start streaming the response to the client at the same time. There's no need to wait on the caching to start sending the response back...

@williaster:

@betodealmeida this is a big change that has the potential to impact many many users. If you're unable to provide numbers that indicate that this is strictly an improvement (or at a minimum no regressions), I'm a bit reluctant to introduce this additional complexity.

I think it's part of the expected work of a feature like this to demonstrate the effects with real-life examples. It seems like you should be able to use real Lyft dashboards, dashboards from the "example datasets", etc., and you can throttle network speed using dev tools.

Sure, I will give the numbers of the example dashboards and some of the Lyft dashboards. My point was that, considering that this is a strict improvement, defining a threshold to accept the changes seems arbitrary to me.
Of course I agree that if this causes a regression or no significant improvement we should not do it. Maybe it's not clear, but with this PR the client will always issue a smaller number of requests, and a percentage of those requests will receive body-less responses.
Also, keep in mind that this PR is against the lyftga branch, so we expect to test it before merging into master.

Another concern I have is the impact on the # of requests. We've needed to introduce domain sharding because of the large number of simultaneous requests made by larger dashboards, and this potential DOUBLES that number, so I would want to see perf numbers that demonstrate no regressions for that case as well.

Sorry, why do you think this would double the number of requests? The number of requests should be strictly equal or smaller: resources within the lifetime of the cache are no longer requested because the browser reads directly from its cache, and conditional requests are still a single request. The server will either return a normal response (200 OK) or a 304 Not Modified without body. There's no pre-flight request.

@john-bodley not sure if I understand your question. Currently we cache the dataframe in viz.BaseViz.get_df_payload. I haven't touched that code, but I added an additional cache storing the Response object instead. This has a few benefits:

It skips unpickling the dataframe and serializing it to json.
We don't need to recompute the ETag every time we read from the cache, since it's stored in the cached response.

Eventually we can remove the dataframe caching, but I'd rather do that in a separate PR.

@graceguo-supercat I changed the code so that in the GET request we pass:

the slice_id
any extra_filters

This way the GET request is still relatively small (and we can switch to Rison for a smaller URL if needed), and the caching mechanism works with dashboards that have filters saved.

@betodealmeida my point was that I felt that the picture and description doesn't accurately reflect what your code is actually doing, i.e., that there's actually two server-side cache keys associated with each chart. They're both using the same underlying cache but the diagram and steps don't accurately reflect this.
I agree with your logic but thought maybe the steps should be more explicit, e.g.,
After

DB returns a dataframe.
Superset caches the dataframe.
Superset serializes the payload.
Superset caches the the HTTP response associated with the payload.
Superset sends the payload to user "A", with an Expires header of 1 hour, and an ETag header which is a hash of the payload.

@john-bodley you're right. I ended up describing in "after" the workflow once we remove the dataframe cache.

Codecov Report

Merging #7032 into lyftga will increase coverage by <.01%.
The diff coverage is 73.2%.

@@            Coverage Diff             @@
##           lyftga    #7032      +/-   ##
==========================================
+ Coverage   64.36%   64.36%   +<.01%
==========================================
  Files         421      422       +1
  Lines       20551    20596      +45
  Branches     2249     2261      +12
==========================================
+ Hits        13228    13257      +29
- Misses       7190     7203      +13
- Partials      133      136       +3

Impacted Files
Coverage Δ

...rset/assets/src/dashboard/containers/Dashboard.jsx
0% <ø> (ø)
:arrow_up:

superset/assets/src/chart/Chart.jsx
13.15% <0%> (-0.74%)
:arrow_down:

...ts/src/explore/components/ExploreViewContainer.jsx
34.04% <0%> (ø)
:arrow_up:

...sets/src/explore/components/ExploreChartHeader.jsx
50% <0%> (ø)
:arrow_up:

...rset/assets/src/dashboard/components/Dashboard.jsx
89.28% <100%> (ø)
:arrow_up:

superset/assets/src/explore/exploreUtils.js
76.47% <30%> (-4.39%)
:arrow_down:

superset/assets/src/chart/chartAction.js
46.21% <70%> (+0.97%)
:arrow_up:

superset/views/core.py
74.45% <70%> (-0.44%)
:arrow_down:

superset/utils/decorators.py
84.78% <81.08%> (-15.22%)
:arrow_down:

superset/views/utils.py
88.23% <86.2%> (-4.87%)
:arrow_down:

... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9855837...27c325b. Read the comment docs.

@xtinec can we merge this into lyftga? It will help test the cache warmup.

@williaster @john-bodley is it ok to merge this? I've addressed all the issues.

@betodealmeida I was wondering whether there was any qualitative or quantitative evidence of any performance improvements?

ditto john's question about something that demonstrates the perf improvements/as I asked earlier. otherwise lgtm 👍

@john-bodley @williaster
Sorry, I ran some numbers and forgot to post them here. I did a few tests with 4 examples dashboards:

For the "Misc Charts", eg, the first run is more than 20% faster, and the second and third requests are more than 50% faster.
Here are the numbers:
https://docs.google.com/spreadsheets/d/1bliQs6fnqni0lcNuxKRQ-A22RhOTiOI6OHZaR2tClh0/edit?usp=sharing

thanks for the benchmarks!
will let @john-bodley sign off since he had the last requested change.

I was working on this tricky bug and ended up here as the cause for it. The issue is around the fact that the formData may need to get sanitized by frontend-related logic.
The bug or symptom is the "Genders by State" example started showing an extra (3rd) metric (sum__num) on top of Boys and Girls.
There's a lot going on related to this, but a key point here is that we have control-related logic like default and things around control panel that sanitizes formData. In that particular case, the example's form data is malformed with both metrics and metric. In the explore view, or in a normal POST request, the formData gets sanitized:

missing keys get filled with the control's default
extra keys get deleted
more stuff

Now another thing that compounds here is that viz.py blindly looks at all METRIC_KEYS to help craft the query_obj. So the combo of these things create the issue:

example is malformed, it should have only metrics and NO metric key (easy to fix)
in GET mode, the formData does not get "sanitized" by the frontend
viz.py could be more prescriptive about the metrics it builds into the query object

Now it's pretty clear that addressing any of these 3 things would fix my symptom, but point 2 on its own is worrisome. It can lead to intricate issues over time. Say I save a chart, and after that I add a new control to that vizType, in the context of the GET, it won't get the right default.
Good news is I'm working on a refactor https://github.com/apache/incubator-superset/pull/7350 to help and clean up all of the control / formData processing logic. It grew out of control over time. Now the assumption is that all request would make it through this logic, and I'm realizing that's not the case, at least since this PR.
Ideas?

Found some other issues here that I wanted to raise ^^^
Also I noticed that the big "merge" on master of this and much more stuff got actually done on a single commit instead of a proper merge 538776b47025f8ff97f293f943736cb3633b2702
In the future, lyftga branches and the likes should be merged, not squashed and merged as we loose tons of history. For instance if we wanted to revert this PR, there's no single commit in master we can address, we'd have to revert all of 538776b47025f8ff97f293f943736cb3633b2702 or get really creative...

@mistercrunch agreed, and after that lyftga branch all of our commits were merged individually (from lyft-release-sp8.  We have now switched to working on and continuously deploying from master so this should not longer even be able to happen. ;-)


chore(package): update rollup to version 0.51.8

Codecov Report

Merging #182 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #182   +/-   ##
======================================
  Coverage    92.3%   92.3%
======================================
  Files           2       2
  Lines          13      13
======================================
  Hits           12      12
  Misses          1       1

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f11ef3...bbd575c. Read the comment docs.

Coverage remained the same at 92.308% when pulling bbd575c6f6d2abe159f50eb9695fe7d48ba28e32 on greenkeeper/rollup-0.51.8 into 7f11ef3362ed82f5025a966598e4bb7e3166ec09 on master.


Update tests to cover replyTo args

Hi!
I didn't meant you to create a new pull request with the tests. The build will never pass, since you are trying to test a new funcionality that is in a different branch :)
Just push this changes to the bartoszzajac:patch-1 branch and the original pull request will get updated.
I'm closing this by the moment.


0.0.359  HelperFile could not find 'stup.exe'

Hello I use the vpk command packaging prompt can not find stup.exe, look at the dotnet tool installation directory did not

After trying to copy the dotnet tool under the setup.exe renamed stup.exe can be

The command line shows as vpk version 0.0.0, it doesn't look right. How did you install vpk?

What was your issue? How did you resolve it?


WIP: Testing update

/retest

/retest

Image pushed and tested.

/retest lint-charts

/test lint-charts

@davidkarlsen This won't help. It's a CircleCI issue.

@unguiculus I was hoping it was just temporarily

/lgtm

/hold

https://github.com/helm/chart-testing/pull/13 causes problem with CircleCI. I already had to add workarounds due to this. We should revert the changes to the Dockerfile. If someone needs this, they should extend the image.

/hold cancel


bug: cassandra concurrent schema updates (initial concurrent table auto-creation)

Cassandra concurrent schema updates (concurrent table auto-creation) lead to schema collisions (tables are created by each task-thread in parallel on first application start).
ℹ As a side note - ScyllaDB shipped 'Strongly Consistent Schema Management with Raft' recently with 5.2 release.
ref: https://www.scylladb.com/2023/05/04/scylladb-open-source-5-2-with-raft-based-schema-management/

The plan to fix/workaround: Always use consistency level ALL for DDL queries.
related:

allow to opt-out of tables to be auto-created #9
allow to set consistency level for DDL via execution profile #11


[ENG-2110] Use custom DOI prefixes for registries providers

Purpose
Currently certain communities have there own special DOI prefixes, and now registries moderation should have it too!
Changes

changes client to send custom prefix
changes tests to work with less mockable client

QA Notes
Please make verification statements inspired by your code and what your code touches.

Verify the datacite test site is getting the custom prefix when set for a registration provider via the admin app.

Documentation
tech task, no new docs.
Side Effects
None that I know of.
Ticket
https://openscience.atlassian.net/browse/ENG-2110

LGTM!


I use gateway Kong Routing is configured

My problem is usering swagger request interface Tips is 404。kong routing is http//xxxx.com/servicename/controllers/aciton. but swagger url is http://xxxx.com/controllers/aciton. How should I set up swagger?

Closing due to inactivity


fix: add check for loaded parent

This addresses a case where the parent group has not been loaded prior to an
update of a child media player.
Closes #496

I've copied the changed code into the media_player.py file and so far I'm not seeing the same errors I was before.
Looks like it was a success!


Added Russian translation

Added Russian translation for :

Research Project
Research Tab
ThingDef
Patches (Chunk, Large Pot, Plinth)

Thanks!


Remove rootCA

I have put demo rootCA into OpenXPKI.
How to remove it and import new rootCA?

In https://github.com/openxpki/openxpki/issues/815 you have been asked to place support requests on the OpenXPKI Users Mailing List. Please do so.
http://www.openxpki.org/support.html


Test failures on Visual Studio 2015 and 2017 using VC-WIN64A-masm build target

Hello again,
We're trying to switch our 64bit builds from VC-WIN64A to VC-WIN64A-masm so we can add the CFG flag (/guard:cf) as described here: https://github.com/openssl/openssl/issues/16147
This report is based on a VS2015 job but we see the same failures on VS2017.
VS2019 and VS2022 are unaffected.
I assume you don't test against the older compilers but they are still supported for now:
https://endoflife.date/visual-studio
19:18:04 Test Summary Report
19:18:04 -------------------
19:18:04 30-test_evp.t                    (Wstat: 512 (exited 2) Tests: 104 Failed: 2)
19:18:04   Failed tests:  4, 36
19:18:04   Non-zero exit status: 2
19:18:04 30-test_evp_libctx.t             (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
19:18:04   Failed test:  1
19:18:04   Non-zero exit status: 1
19:18:04 80-test_cms.t                    (Wstat: 512 (exited 2) Tests: 17 Failed: 2)
19:18:04   Failed tests:  4-5
19:18:04   Non-zero exit status: 2
19:18:04 Files=250, Tests=3303, 729 wallclock secs ( 4.00 usr +  0.73 sys =  4.74 CPU)
19:18:04 Result: FAIL

I'm attaching the full verbose test logs as a file since I'm having trouble filtering out exactly what's needed.
Sorry about that but I hope you can make more sense of it than me :)
windows-masm-test-failures-verbose.txt
Here's the buildconfig:
buildconfig_md.txt
But please note that we modify ASFLAGS, CFLAGS and LDFLAGS in the generated makefile since the configure script does not allow us to mix configure options and make flags, so this is what's built:
##### User defined commands and flags ################################

CC="cl"
CPP=$(CC) /EP /C
CPPFLAGS=
CFLAGS=/sdl /Qspectre /guard:cf /W3 /wd4090 /nologo /O2
LD="link"
LDFLAGS=/CETCOMPAT /guard:cf /nologo /debug
EX_LIBS=

PERL=C:\Strawberry\perl\bin\perl.exe

AR="lib"
ARFLAGS= /nologo

MT="mt"
MTFLAGS= -nologo

AS="ml64"
ASFLAGS=-g /nologo /Zi

RC="rc"
RCFLAGS=

ECHO="$(PERL)" "$(SRCDIR)\util\echo.pl"

We've done it this way for a long time and haven't ran into issues but I wonder if there's an officially supported way to configure these extra flags as it does seem a bit brittle as is.
Thanks in advance!

Looking at the first test failure:
C:\Strawberry\perl\bin\perl.exe ..\..\util\wrap.pl ..\..\test\evp_test.exe -config ..\..\test\default-and-legacy.cnf ..\..\test\recipes\30-test_evp_data\evppkey_sm2.txt => 0
15:11:21 ok 101 - running evp_test -config ..\..\test\default-and-legacy.cnf evppkey_sm2.txt
15:11:21 ok 102 # skip DSA not disabled or ERR disabled
15:11:21 ok 103 # skip DSA not disabled or ERR disabled
15:11:21 ok 104 # skip SM2 not disabled
15:11:21 # Looks like you failed 2 tests of 104.
15:11:21 Dubious, test returned 2 (wstat 512, 0x200)
15:11:21 Failed 2/104 subtests
15:11:21 	(less 3 skipped subtests: 99 okay)

It looks like evp_test.exe exited with an exit code of 2, which is Dubious, as the expectation is an exit of only 1 or 0.
As to why that happened, its hard to say from these logs.  Usually when I hit that, I run the test by hand to inspect it further, in this case you would run (from the root directory of the source tree):
.\test\evp_test.exe -config .\test\default-and-legacy.cnf .\test\recipes\30-test_evp_data\evppkey_sm2.txt

Note, you may need to adjust your path to find libcrypto.dll and libssl.dll in the current directory, but I think windows searches the current directory first
That will give you some more visibility into the issue.  Depending on what output is produced, it may require running under a debugger to further root cause the issue

They pass when run manually on a build machine held for debugging..
The .dlls are in the same directory I'm running from, however I've noticed that this new assembler changes the binary names on 64bit from libcrypto-3-x64.dll to libcrypto-3.dll and libssl-3-x64.dll to libssl-3.dll.
This is something we need to account for since our end users expect the former naming convention, which has been the default output for all our previous releases.
Here's the output from that command:
C:\jk\p4\ThirdParty\openssl\3.0\_src\openssl-3.0-src\openssl-3.0>.\test\evp_test
.exe -config .\test\default-and-legacy.cnf .\test\recipes\30-test_evp_data\evppk
ey_sm2.txt
1..1
    # Subtest: run_file_tests
    1..1
    # INFO:
    # Reading .\test\recipes\30-test_evp_data\evppkey_sm2.txt
    # INFO:
    # Starting "SM2 tests" tests at line 19
    # INFO:
    # Starting "SM2 key generation tests" tests at line 78
    # INFO:
    # Completed 10 tests with 0 errors and 0 skipped
    ok 1 - iteration 1
ok 1 - run_file_tests

And another one I took a guess at (also tried using default-and-legacy.cnf - same results):
C:\jk\p4\ThirdParty\openssl\3.0\_src\openssl-3.0-src\openssl-3.0>.\test\evp_libc
tx_test.exe -config .\test\default.cnf
1..9
    # Subtest: test_dsa_param_keygen
    1..27
    ok 1 - iteration 1
    ok 2 - iteration 2
    ok 3 - iteration 3
    ok 4 - iteration 4
    ok 5 - iteration 5
    ok 6 - iteration 6
    ok 7 - iteration 7
    ok 8 - iteration 8
    ok 9 - iteration 9
    ok 10 - iteration 10
    ok 11 - iteration 11
    ok 12 - iteration 12
    ok 13 - iteration 13
    ok 14 - iteration 14
    ok 15 - iteration 15
    ok 16 - iteration 16
    ok 17 - iteration 17
    ok 18 - iteration 18
    ok 19 - iteration 19
    ok 20 - iteration 20
    ok 21 - iteration 21
    ok 22 - iteration 22
    ok 23 - iteration 23
    ok 24 - iteration 24
    ok 25 - iteration 25
    ok 26 - iteration 26
    ok 27 - iteration 27
ok 1 - test_dsa_param_keygen
    # Subtest: test_dh_safeprime_param_keygen
    1..27
    ok 28 - iteration 1
    ok 29 - iteration 2
    ok 30 - iteration 3
    ok 31 - iteration 4
    ok 32 - iteration 5
    ok 33 - iteration 6
    ok 34 - iteration 7
    ok 35 - iteration 8
    ok 36 - iteration 9
    ok 37 - iteration 10
    ok 38 - iteration 11
    ok 39 - iteration 12
    ok 40 - iteration 13
    ok 41 - iteration 14
    ok 42 - iteration 15
    ok 43 - iteration 16
    ok 44 - iteration 17
    ok 45 - iteration 18
    ok 46 - iteration 19
    ok 47 - iteration 20
    ok 48 - iteration 21
    ok 49 - iteration 22
    ok 50 - iteration 23
    ok 51 - iteration 24
    ok 52 - iteration 25
    ok 53 - iteration 26
    ok 54 - iteration 27
ok 2 - test_dh_safeprime_param_keygen
ok 3 - dhx_cert_load
    # Subtest: test_cipher_reinit
    1..124
    ok 55 - iteration 1
    ok 56 - iteration 2
    ok 57 - iteration 3
    ok 58 - iteration 4
    ok 59 - iteration 5
    ok 60 - iteration 6
    ok 61 - iteration 7
    ok 62 - iteration 8
    ok 63 - iteration 9
    ok 64 - iteration 10
    ok 65 - iteration 11
    ok 66 - iteration 12
    ok 67 - iteration 13
    ok 68 - iteration 14
    ok 69 - iteration 15
    ok 70 - iteration 16
    ok 71 - iteration 17
    ok 72 - iteration 18
    ok 73 - iteration 19
    ok 74 - iteration 20
    ok 75 - iteration 21
    ok 76 - iteration 22
    ok 77 - iteration 23
    ok 78 - iteration 24
    ok 79 - iteration 25
    ok 80 - iteration 26
    ok 81 - iteration 27
    ok 82 - iteration 28
    ok 83 - iteration 29
    ok 84 - iteration 30
    ok 85 - iteration 31
    ok 86 - iteration 32
    ok 87 - iteration 33
    ok 88 - iteration 34
    ok 89 - iteration 35
    ok 90 - iteration 36
    ok 91 - iteration 37
    ok 92 - iteration 38
    ok 93 - iteration 39
    ok 94 - iteration 40
    ok 95 - iteration 41
    ok 96 - iteration 42
    ok 97 - iteration 43
    ok 98 - iteration 44
    ok 99 - iteration 45
    ok 100 - iteration 46
    ok 101 - iteration 47
    ok 102 - iteration 48
    ok 103 - iteration 49
    ok 104 - iteration 50
    ok 105 - iteration 51
    ok 106 - iteration 52
    ok 107 - iteration 53
    ok 108 - iteration 54
    ok 109 - iteration 55
    ok 110 - iteration 56
    ok 111 - iteration 57
    ok 112 - iteration 58
    ok 113 - iteration 59
    ok 114 - iteration 60
    ok 115 - iteration 61
    ok 116 - iteration 62
    ok 117 - iteration 63
    ok 118 - iteration 64
    ok 119 - iteration 65
    ok 120 - iteration 66
    ok 121 - iteration 67
    ok 122 - iteration 68
    ok 123 - iteration 69
    ok 124 - iteration 70
    ok 125 - iteration 71
    ok 126 - iteration 72
    ok 127 - iteration 73
    ok 128 - iteration 74
    ok 129 - iteration 75
    ok 130 - iteration 76
    ok 131 - iteration 77
    ok 132 - iteration 78
    ok 133 - iteration 79
    ok 134 - iteration 80
    ok 135 - iteration 81
    ok 136 - iteration 82
    ok 137 - iteration 83
    ok 138 - iteration 84
    ok 139 - iteration 85
    ok 140 - iteration 86
    ok 141 - iteration 87
    ok 142 - iteration 88
    ok 143 - iteration 89
    ok 144 - iteration 90
    ok 145 - iteration 91
    ok 146 - iteration 92
    ok 147 - iteration 93
    ok 148 - iteration 94
    ok 149 - iteration 95
    ok 150 - iteration 96
    ok 151 - iteration 97
    ok 152 - iteration 98
    ok 153 - iteration 99
    ok 154 - iteration 100
    ok 155 - iteration 101
    ok 156 - iteration 102
    ok 157 - iteration 103
    ok 158 - iteration 104
    ok 159 - iteration 105
    ok 160 - iteration 106
    ok 161 - iteration 107
    ok 162 - iteration 108
    ok 163 - iteration 109
    ok 164 - iteration 110
    ok 165 - iteration 111
    ok 166 - iteration 112
    ok 167 - iteration 113
    ok 168 - iteration 114
    ok 169 - iteration 115
    ok 170 - iteration 116
    ok 171 - iteration 117
    ok 172 - iteration 118
    ok 173 - iteration 119
    ok 174 - iteration 120
    ok 175 - iteration 121
    ok 176 - iteration 122
    ok 177 - iteration 123
    ok 178 - iteration 124
ok 4 - test_cipher_reinit
    # Subtest: test_cipher_reinit_partialupdate
    1..124
    ok 179 - iteration 1
    ok 180 - iteration 2
    ok 181 - iteration 3
    ok 182 - iteration 4
    ok 183 - iteration 5
    ok 184 - iteration 6
    ok 185 - iteration 7
    ok 186 - iteration 8
    ok 187 - iteration 9
    ok 188 - iteration 10
    ok 189 - iteration 11
    ok 190 - iteration 12
    ok 191 - iteration 13
    ok 192 - iteration 14
    ok 193 - iteration 15
    ok 194 - iteration 16
    ok 195 - iteration 17
    ok 196 - iteration 18
    ok 197 - iteration 19
    ok 198 - iteration 20
    ok 199 - iteration 21
    ok 200 - iteration 22
    ok 201 - iteration 23
    ok 202 - iteration 24
    ok 203 - iteration 25
    ok 204 - iteration 26
    ok 205 - iteration 27
    ok 206 - iteration 28
    ok 207 - iteration 29
    ok 208 - iteration 30
    ok 209 - iteration 31
    ok 210 - iteration 32
    ok 211 - iteration 33
    ok 212 - iteration 34
    ok 213 - iteration 35
    ok 214 - iteration 36
    ok 215 - iteration 37
    ok 216 - iteration 38
    ok 217 - iteration 39
    ok 218 - iteration 40
    ok 219 - iteration 41
    ok 220 - iteration 42
    ok 221 - iteration 43
    ok 222 - iteration 44
    ok 223 - iteration 45
    ok 224 - iteration 46
    ok 225 - iteration 47
    ok 226 - iteration 48
    ok 227 - iteration 49
    ok 228 - iteration 50
    ok 229 - iteration 51
    ok 230 - iteration 52
    ok 231 - iteration 53
    ok 232 - iteration 54
    ok 233 - iteration 55
    ok 234 - iteration 56
    ok 235 - iteration 57
    ok 236 - iteration 58
    ok 237 - iteration 59
    ok 238 - iteration 60
    ok 239 - iteration 61
    ok 240 - iteration 62
    ok 241 - iteration 63
    ok 242 - iteration 64
    ok 243 - iteration 65
    ok 244 - iteration 66
    ok 245 - iteration 67
    ok 246 - iteration 68
    ok 247 - iteration 69
    ok 248 - iteration 70
    ok 249 - iteration 71
    ok 250 - iteration 72
    ok 251 - iteration 73
    ok 252 - iteration 74
    ok 253 - iteration 75
    ok 254 - iteration 76
    ok 255 - iteration 77
    ok 256 - iteration 78
    ok 257 - iteration 79
    ok 258 - iteration 80
    ok 259 - iteration 81
    ok 260 - iteration 82
    ok 261 - iteration 83
    ok 262 - iteration 84
    ok 263 - iteration 85
    ok 264 - iteration 86
    ok 265 - iteration 87
    ok 266 - iteration 88
    ok 267 - iteration 89
    ok 268 - iteration 90
    ok 269 - iteration 91
    ok 270 - iteration 92
    ok 271 - iteration 93
    ok 272 - iteration 94
    ok 273 - iteration 95
    ok 274 - iteration 96
    ok 275 - iteration 97
    ok 276 - iteration 98
    ok 277 - iteration 99
    ok 278 - iteration 100
    ok 279 - iteration 101
    ok 280 - iteration 102
    ok 281 - iteration 103
    ok 282 - iteration 104
    ok 283 - iteration 105
    ok 284 - iteration 106
    ok 285 - iteration 107
    ok 286 - iteration 108
    ok 287 - iteration 109
    ok 288 - iteration 110
    ok 289 - iteration 111
    ok 290 - iteration 112
    ok 291 - iteration 113
    ok 292 - iteration 114
    ok 293 - iteration 115
    ok 294 - iteration 116
    ok 295 - iteration 117
    ok 296 - iteration 118
    ok 297 - iteration 119
    ok 298 - iteration 120
    ok 299 - iteration 121
    ok 300 - iteration 122
    ok 301 - iteration 123
    ok 302 - iteration 124
ok 5 - test_cipher_reinit_partialupdate
ok 6 - kem_rsa_gen_recover
ok 7 - kem_rsa_params
ok 8 - kem_invalid_keytype
ok 9 - test_cipher_tdes_randkey

Any ideas what's gone wrong?

I don't immediately see anything that would have modified the output dlls from anything other than lib[crypto|ssl]-<version.dll.  @jamuir do you see anything that might have done that?
As for your error (or lack of it ) when running locally, It appears as though the test plan file (30-test_evp.t) pulls in several other config files (though its still odd, given that the report says it failed on the sm2 tests).
If you just run this locally:
make TESTS=test_evp V=1 test

Does it still succeed

That crashes when I run it. The debugger shows this exception for both dubious failures..
C:\jk\p4\ThirdParty\openssl\3.0\_src\openssl-3.0-src\openssl-3.0>call C:\"Program Files (x86)"\"Microsoft Visual Studio 14.0"\VC\bin\amd64\vcvars64.bat&&nmake test TESTS=test_evp V=1
...
Unhandled exception at 0x00007FFC03061178 (ntdll.dll) in evp_test.exe: RangeChecks instrumentation code detected an out of range array access.

Unfortunately this build agent has some VS licensing issues so I only have screenshots right now..
test_evp
First failure:

Second failure:

I also ran into the same issue with the other test suites.
test_evp_libctx

test_cms

I don't immediately see anything that would have modified the output dlls from anything other than lib[crypto|ssl]-<version.dll. @jamuir do you see anything that might have done that?

I have no experience with masm.  I took a look at the log, but nothing jumped out at me.
Based on the following comment in 50-masm.conf, I think some issues with older masm versions were anticipated:
https://github.com/openssl/openssl/blob/143ca66cf00c88950d689a8aa0c89888052669f4/Configurations/50-masm.conf#L1-L8
@robmcgee-mag :  do these failures appear after building in a completely clean tree?  When changing/updating windows configs, it is really important to start with a clean tree (c.f. https://github.com/openssl/openssl/discussions/22385 )
Regarding the exception message:

Unhandled exception at 0x00007FFC03061178 (ntdll.dll) in evp_test.exe: RangeChecks instrumentation code detected an out of range array access.

if you drop the flag /guard:cf, then do the failures still happen?

The guard flag is whats catching the error (From what I read its roughly the equivalent of libasans memcheck)
@robmcgee-mag Thank you for the screenshots, but its a bit tough from them to determine where exactly the error is.  If I'm reading it correctly, I think its this line:
memcpy(R, B + 8, 8);

thats triggering the error, is that correct?
Its a bit early here, so I'm not 100% sure if I'm reading this correctly, and I'm even less sure how this wouldn't have been caught on the asan runs in ci, but in this code B is defined as unsigned char B[16] on the stack.  By my read both of those memcpy operations to and from B should overrun the array by one.  Copying to B+8 for a length of 8 should write the last byte to array index 16, which appears to me to be off by one.  Please check me on that.
That just doesn't make any sense to me, but if its the case, I think the fix would be changing the copy length to 7 in those memcpys

Copying to B+8 for a length of 8

Err, I'm not sure about that.
1 byte would be written to B[8], a second to B[9], the nth to B[8 + n - 1], so the 8th byte to B[15], which is within the bounds of an array declared unsigned char B[16] (valid accesses are B[0] to B[15]

I'm definitely not sure about it either, I'm just trying to puzzle through whats going on.  By my read:
memcpy(R, B + 8, 8);

Is a copy from B (unsigned char B[16] to R), so
B[0] = B
B[1] = B +1
B[2] = B + 2
B[3] = B + 3
B[4] = B + 4
B[5] = B + 5
B[6] = B + 6
B[7] = B + 7
B[8] = B + 8       <= read 1 byte here (1)
B[9] = B + 9       <= read 1 byte here (2)
B[10] = B + 10   <= read 1 byte here (3)
B[11] = B + 11   <= read 1 byte here (4)
B[12] = B + 12   <= read 1 byte here (5)
B[13] = B + 13   <= read 1 byte here (6)
B{14] = B + 14   <= read 1 byte here (7)
B[15] = B + 15   <= read 1 byte here (8)
So yeah, you're right, thats totally in bounds
But it raises the question, whats going on here?  False positive in the masm address sanitizer?
Actually, @robmcgee-mag something else looks odd to me, you're first screenshot shows this code:
memcpy(R, B + 8, 8);
        }
    }
memcpy(out, A, 8);
return inlen + 8;

The stack frame listed in that image is crypto_128_unwrap_raw, but that code doesn't appear there, it appears in CRYPTO_128_wrap (i.e. not CRYPTO_128_unwrap).  So the code we're looking at here doesn't match the stack frame listed.  Can you go and make sure the right code is displayed in the debugger please?

@jamuir Our build agents are ephemeral machines that spin up a known-good image on request and sync all code needed to build, so I would say they're about as clean as can be.
However the examples in the last comment I provided were from my own manual tests on one of those machines after the automation had already failed on nmake test.
I would still consider this clean and I've also reproduced the failures on my local machine with a fresh install of VS2017, albeit without the exception and debugger prompt appearing, which I'm still looking into..
I do still see these failures without the /guard:cf flag so it seems the assembler itself is at fault.
https://github.com/openssl/openssl/blob/143ca66cf00c88950d689a8aa0c89888052669f4/Configurations/50-masm.conf#L1-L8
From the sound of those docs, CFG via masm isn't something that you fully support on all VS compilers, right?

I don't immediately see anything that would have modified the output dlls from anything other than lib[crypto|ssl]-<version.dll.

The 64bit naming convention we have for all 3.0.X builds and 1.1.1 builds also had "x64" in it so I'm confused how you don't have the same..

thats....odd.  My understanding of /guard:cf was that it added various code/data guard checks (effectively the masm version of libasan), and these errors appear (at least to me) to be guard check failures, so I'm at a bit of a loss as to how to explain how one would get guard failures in the code without adding guard checks.
you are correct, we don't support masm, for the reasons stated above, but my understanding is that masm versions change their usage on the command line between versions, and so we can't setup a build environment that works with each one.  But this doesn't seem to be that kind of problem.  The library built fine, its just catching an error at run time


[3.2] Fix specular render_mode for visual shaders

Fix https://github.com/godotengine/godot/issues/41525 for 3.2

Thanks!


Font Request: Solus logo

Solus is a linux distribution.
The logo is available in this archive /solus/SVG/Solus.svg
Thanks :smiley:
CC: @JoshStrobl in case there should be a need of a modified SVG logo

Added via 82a9a6ed7d7a38ffc5c36ac349fb94127b5563ed.


Refactor UI and make it easy for analysts to use it.

@essamjoubori @mgrauer

@essamjoubori @katelynnobrien and I will come up some specific next steps.
The general idea here is that we want to have some of the power of a GIS type application, but allow usage by someone who is not a GIS expert or power user, so we will offer limited options.


Project: For stakeholders summarized Azure Cognitive Service with Graph .net SDK #Hack

Project name
stakeholders summarized Azure Cognitive Service with Graph .net SDK
Description
Real Time Problem:
As my CTO having responsibility to look after multiple projects and we as developers having multiple meeting internal external. It was challenging for him to listen or understand complete scenarios or read lengthy emails about discussion.
Solution:
Using Microsoft Graph .net SDK i have created web application using ,net 7.0 for Team leads to see all emails of belongs to his project.

Using Azure Cognitive Service summarizing email.
From Email tab emails Team leads can select specific email.
On dashboard Team lead can select Microsoft specific Teams and specific channels.
From Same view team leads can post summarized discussion to teams channel.

SO higher management or CTO can understand what going on rather then reading complete email.
Configuration Details for Azure App:
dotnet user-secrets init
dotnet user-secrets set "AzureAd:ClientId" " "
dotnet user-secrets set "AzureAd:ClientSecret" " "

Appsettings.json update following configuration for cognitive service application:
"ConfigurationAzure": {
   "credentials": "",
   "endpoint": ""
 }

still working on it to enhance. Getting other ideas as well.
Repo URL
https://github.com/sabeehsharif/MicrosoftGraph-HackTogether
Team members
Later

@waldekmastykarz  Thanks you can view and download the video using this one drive public Link Summarized Graph Hack


Bug: Text in edit control is too small on scaled 4K display

Fixed in https://github.com/matthewjustice/essential-notepad/commit/d0e64fc387468ed785782325e2d0dd9584909648


Short solution needed: "Save dictionary to Redis" (python-redis)

Please help us write most modern and shortest code solution for this issue:
Save dictionary to Redis (technology: python-redis)
Fast way
Just write the code solution in the comments.
Prefered way

Create pull request with a new code file inside inbox folder.
Don't forget to use comments to make solution explained.
Link to this issue in comments of pull request.

Save dict to Redis


Fullscreen with multi-screen works not properly

I have 2 screens, one is 19201080, another is 38402160.
the first screen coordinates is [-1920,0],[0,1080],
and the second is [0,3840],[0,2160]
Now the full screen position is [-1920,1920],[0,2160]

--fullscreen is supposed to fill one of the monitors, the one the window isi currently in. if you want a background that spans both monitors you will have to set coordinates manually with the move command for now. maybe i'll consider adding another kind of fullscreen, like maybe --multimonitor or something that spans all monitors

Thanks.
But it seems --fullscreen does not behavior as expected. Wherever I put the window, fullscreen will fill 1080p screen with 2160p size.

hm, interesting, I remember testing on my dual monitor (1280x1024 and 1920x1080) and it was working as expected, but I guess i'll re-test --fullscreen as well

ok i see the problem with --fullscreen, the coordinates used in wp mv seem to be always positive even on multimonitor setup, which doesn't match the coordinates returned when i query the monitors (for example my 2nd monitor is monitor: -1280 0 0 1024). i'm currently testing if MapWindowPoints from the desktop window to the root window is what i want and it seems like it behaves as it should
but there's another issue, for some reason it always wants to put it on my 2nd monitor, so maybe I'll use the cursor position to figure out which monitor instead of using MonitorFromWindow which doesn't seem to work

ok i fixed it and also added a --panoramic option to make a window fill all monitors


Funimation Error 4 No premium avails, but videos are available on website

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.02.04. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[x] I've read about sharing account credentials and I'm willing to share it if required

Region
United Kingdom
Description
I was doing a batch series download when I noticed that some episodes in the same series and season were missing. Went back through the console and found issues with the English dub format not being available, Japanese is fine just English on particular episodes.

Video runs fine on the actual website.
I do have a premium account and have had this same error with different episodes from this series and others.
Verbose log
[debug] Command-line config: ['-vU', '-o', '%(series)s-S%(season_number)sE%(episode_number)s-%(episode)s.%(ext)s', '--no-check-certificate', '--netrc', '--cookies', 'funimation.com_cookies.txt', 'http://www.funimation.com/v/a-certain-magical-index/remnant-wreckage', '--extractor-args', 'funimation:language=english']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, err UTF-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [c1653e9ef]
[debug] Python version 3.7.3 (CPython 32bit) - Linux-5.10.63-v7l+-armv7l-with-debian-10.11
[debug] exe versions: ffmpeg 4.1.8-0, ffprobe 4.1.8-0
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.02.04, Current version: 2022.02.04
yt-dlp is up to date (2022.02.04)
[funimation:page] Logging in
[debug] [funimation:page] Extracting URL: http://www.funimation.com/v/a-certain-magical-index/remnant-wreckage
[funimation:page] a-certain-magical-index_remnant-wreckage: Downloading JSON metadata
[debug] [Funimation] Extracting URL: https://www.funimation.com/player/609825
[Funimation] 609825: Downloading player webpage for 609825
[Funimation] remnant-wreckage: Downloading Uncut english (609825) JSON
WARNING: [Funimation] Funimation said: Error 4 - No valid Premium Avails for this experience
ERROR: [Funimation] 93545: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp , filling out the "Broken site" issue template properly. Confirm you are on the latest version using -U
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1381, in wrapper
    return func(self, *args, **kwargs)
  File "/home/pi/.local/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1465, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/pi/.local/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1517, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/home/pi/.local/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2431, in process_video_result
    self.raise_no_formats(info_dict)
  File "/home/pi/.local/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 962, in raise_no_formats
    expected=has_drm or expected)
yt_dlp.utils.ExtractorError: [Funimation] 93545: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp , filling out the "Broken site" issue template properly. Confirm you are on the latest version using -U

EDIT: Forgot to add, when i do --list formats i get this:
[debug] Command-line config: ['-vU', '-o', '%(series)s-S%(season_number)sE%(episode_number)s-%(episode)s.%(ext)s', '--no-check-certificate', '--netrc', '--cookies', 'funimation.com_cookies.txt', 'http://www.funimation.com/v/a-certain-magical-index/remnant-wreckage', '--list-formats']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, err UTF-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [https://github.com/yt-dlp/yt-dlp/commit/c1653e9efba2768910e6a5f62c88c7a95317b431]
[debug] Python version 3.7.3 (CPython 32bit) - Linux-5.10.63-v7l+-armv7l-with-debian-10.11
[debug] exe versions: ffmpeg 4.1.8-0, ffprobe 4.1.8-0
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.02.04, Current version: 2022.02.04
yt-dlp is up to date (2022.02.04)
[funimation:page] Logging in
[debug] [funimation:page] Extracting URL: http://www.funimation.com/v/a-certain-magical-index/remnant-wreckage
[funimation:page] a-certain-magical-index_remnant-wreckage: Downloading JSON metadata
[debug] [Funimation] Extracting URL: https://www.funimation.com/player/609825
[Funimation] 609825: Downloading player webpage for 609825
[Funimation] remnant-wreckage: Downloading Uncut japanese (982216) JSON
[Funimation] remnant-wreckage: Downloading Uncut japanese (982216) m3u8 information
[Funimation] remnant-wreckage: Downloading Uncut english (609825) JSON
WARNING: [Funimation] Funimation said: Error 4 - No valid Premium Avails for this experience
[debug] Sort order given by extractor: lang, source
[debug] Formats sorted by: hasvid, ie_pref, lang, source, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, id
[info] Available formats for 93545:
ID EXT RESOLUTION FPS │ FILESIZE TBR PROTO │ VCODEC VBR ACODEC MORE INFO
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
982216-hls-audio-aacl-256-Japanese mp4 audio only │ m3u8_n │ audio only unknown [japanese] Uncut
982216-mp4 mp4 unknown │ https │ unknown unknown [japanese] Uncut
982216-hls-416 mp4 416x234 23 │ ~ 73.68MiB 416k m3u8_n │ avc1.4D4028 416k video only [japanese] Uncut
982216-hls-574 mp4 640x360 23 │ ~101.67MiB 574k m3u8_n │ avc1.4D4028 574k video only [japanese] Uncut
982216-hls-849 mp4 768x432 23 │ ~150.38MiB 849k m3u8_n │ avc1.4D4028 849k video only [japanese] Uncut
982216-hls-1157 mp4 768x432 23 │ ~204.93MiB 1157k m3u8_n │ avc1.4D4028 1157k video only [japanese] Uncut
982216-hls-1825 mp4 960x540 23 │ ~323.25MiB 1825k m3u8_n │ avc1.4D4028 1825k video only [japanese] Uncut
982216-hls-2553 mp4 1280x720 23 │ ~452.20MiB 2553k m3u8_n │ avc1.640028 2553k video only [japanese] Uncut
982216-hls-3842 mp4 1280x720 23 │ ~680.51MiB 3842k m3u8_n │ avc1.640028 3842k video only [japanese] Uncut
982216-hls-4919 mp4 1920x1080 23 │ ~871.27MiB 4919k m3u8_n │ avc1.640028 4919k video only [japanese] Uncut
982216-hls-6497 mp4 1920x1080 23 │ ~ 1.12GiB 6497k m3u8_n │ avc1.640028 6497k video only [japanese] Uncut

Forgot to add, when i do --list formats i get this:
[debug] Command-line config: ['-vU', '-o', '%(series)s-S%(season_number)sE%(episode_number)s-%(episode)s.%(ext)s', '--no-check-certificate', '--netrc', '--cookies', 'funimation.com_cookies.txt', 'http://www.funimation.com/v/a-certain-magical-index/remnant-wreckage', '--list-formats']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, err UTF-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [c1653e9ef]
[debug] Python version 3.7.3 (CPython 32bit) - Linux-5.10.63-v7l+-armv7l-with-debian-10.11
[debug] exe versions: ffmpeg 4.1.8-0, ffprobe 4.1.8-0
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.02.04, Current version: 2022.02.04
yt-dlp is up to date (2022.02.04)
[funimation:page] Logging in
[debug] [funimation:page] Extracting URL: http://www.funimation.com/v/a-certain-magical-index/remnant-wreckage
[funimation:page] a-certain-magical-index_remnant-wreckage: Downloading JSON metadata
[debug] [Funimation] Extracting URL: https://www.funimation.com/player/609825
[Funimation] 609825: Downloading player webpage for 609825
[Funimation] remnant-wreckage: Downloading Uncut japanese (982216) JSON
[Funimation] remnant-wreckage: Downloading Uncut japanese (982216) m3u8 information
[Funimation] remnant-wreckage: Downloading Uncut english (609825) JSON
WARNING: [Funimation] Funimation said: Error 4 - No valid Premium Avails for this experience
[debug] Sort order given by extractor: lang, source
[debug] Formats sorted by: hasvid, ie_pref, lang, source, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, id
[info] Available formats for 93545:
ID                                 EXT RESOLUTION FPS │   FILESIZE   TBR PROTO  │ VCODEC        VBR ACODEC     MORE INFO
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
982216-hls-audio-aacl-256-Japanese mp4 audio only     │                  m3u8_n │ audio only        unknown    [japanese] Uncut
982216-mp4                         mp4 unknown        │                  https  │ unknown           unknown    [japanese] Uncut
982216-hls-416                     mp4 416x234     23 │ ~ 73.68MiB  416k m3u8_n │ avc1.4D4028  416k video only [japanese] Uncut
982216-hls-574                     mp4 640x360     23 │ ~101.67MiB  574k m3u8_n │ avc1.4D4028  574k video only [japanese] Uncut
982216-hls-849                     mp4 768x432     23 │ ~150.38MiB  849k m3u8_n │ avc1.4D4028  849k video only [japanese] Uncut
982216-hls-1157                    mp4 768x432     23 │ ~204.93MiB 1157k m3u8_n │ avc1.4D4028 1157k video only [japanese] Uncut
982216-hls-1825                    mp4 960x540     23 │ ~323.25MiB 1825k m3u8_n │ avc1.4D4028 1825k video only [japanese] Uncut
982216-hls-2553                    mp4 1280x720    23 │ ~452.20MiB 2553k m3u8_n │ avc1.640028 2553k video only [japanese] Uncut
982216-hls-3842                    mp4 1280x720    23 │ ~680.51MiB 3842k m3u8_n │ avc1.640028 3842k video only [japanese] Uncut
982216-hls-4919                    mp4 1920x1080   23 │ ~871.27MiB 4919k m3u8_n │ avc1.640028 4919k video only [japanese] Uncut
982216-hls-6497                    mp4 1920x1080   23 │ ~  1.12GiB 6497k m3u8_n │ avc1.640028 6497k video only [japanese] Uncut

I'm unsure if my issue is related but i'm currently having issues downloading the correct language of a show where that language required premium (i do have a premium membership) it happened at around 9pm yesterday when i was in the middle of downloaing a show. got the forst 3 episodes in English and the rest in Japanese and now i can only get JP to work

Try with only cookies and no netrc
Related: https://github.com/yt-dlp/yt-dlp/issues/1569

Follow up, logged into funimation with my account and accessed the player page directly with this: https://www.funimation.com/player/609825
which resulted in the following screen

So this may be an issue on funimations end, but i have no idea really.
The weird thing is, is that the video plays fine on the funimation website without hitting the player page directly.
Any thoughts?

Most likely same underlying issue as https://github.com/yt-dlp/yt-dlp/issues/2255
The whole extractor looks like it needs rewriting

Seems like the problem has kind of resolved its self:
It Looks like funimation are currently re-jiggling their database, note the player ids are different this time compared to last. So my advice to others with similar issues, give it 3-4 days it seems to have worked for me
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, err UTF-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [c1653e9ef]
[debug] Python version 3.7.3 (CPython 32bit) - Linux-5.10.63-v7l+-armv7l-with-debian-10.11
[debug] exe versions: ffmpeg 4.1.8-0, ffprobe 4.1.8-0
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.02.04, Current version: 2022.02.04
yt-dlp is up to date (2022.02.04)
[debug] [funimation:page] Extracting URL: https://www.funimation.com/en/shows/a-certain-magical-index/remnant-wreckage/
[funimation:page] a-certain-magical-index_remnant-wreckage: Downloading JSON metadata
[debug] [Funimation] Extracting URL: https://www.funimation.com/player/982250
[Funimation] 982250: Downloading player webpage for 982250
[Funimation] remnant-wreckage: Downloading Uncut english (982250) JSON
[Funimation] remnant-wreckage: Downloading Uncut english (982250) m3u8 information
[debug] Sort order given by extractor: lang, source
[debug] Formats sorted by: hasvid, ie_pref, lang, source, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, id
[info] Available formats for 93545:
ID                                EXT RESOLUTION FPS │   FILESIZE   TBR PROTO  │ VCODEC        VBR ACODEC     MORE INFO
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
982250-hls-audio-aacl-256-English mp4 audio only     │                  m3u8_n │ audio only        unknown    [english] Uncut
982250-mp4                        mp4 unknown        │                  https  │ unknown           unknown    [english] Uncut
982250-hls-416                    mp4 416x234     23 │ ~ 73.68MiB  416k m3u8_n │ avc1.4D4028  416k video only [english] Uncut
982250-hls-577                    mp4 640x360     23 │ ~102.20MiB  577k m3u8_n │ avc1.4D4028  577k video only [english] Uncut
982250-hls-848                    mp4 768x432     23 │ ~150.20MiB  848k m3u8_n │ avc1.4D4028  848k video only [english] Uncut
982250-hls-1154                   mp4 768x432     23 │ ~204.40MiB 1154k m3u8_n │ avc1.4D4028 1154k video only [english] Uncut
982250-hls-1830                   mp4 960x540     23 │ ~324.14MiB 1830k m3u8_n │ avc1.4D4028 1830k video only [english] Uncut
982250-hls-2539                   mp4 1280x720    23 │ ~449.72MiB 2539k m3u8_n │ avc1.640028 2539k video only [english] Uncut
982250-hls-3814                   mp4 1280x720    23 │ ~675.55MiB 3814k m3u8_n │ avc1.640028 3814k video only [english] Uncut
982250-hls-4756                   mp4 1920x1080   23 │ ~842.40MiB 4756k m3u8_n │ avc1.640028 4756k video only [english] Uncut
982250-hls-6461                   mp4 1920x1080   23 │ ~  1.12GiB 6461k m3u8_n │ avc1.640028 6461k video only [english] Uncut

Seems like the problem has kind of resolved its self: It Looks like funimation are currently re-jiggling their database, note the player ids are different this time compared to last.

Doubt it, it's more likely that since you were using outdated cookies it was trying to serve you simulcast (since Uncut is usually members only) before, once you retried with fresh cookies it was able to serve you uncut (and hence a different player url).

Seems like the problem has kind of resolved its self: It Looks like funimation are currently re-jiggling their database, note the player ids are different this time compared to last.

Doubt it, it's more likely that since you were using outdated cookies it was trying to serve you simulcast (since Uncut is usually members only) before, once you retried with fresh cookies it was able to serve you uncut (and hence a different player url).

When i first did it i did refresh the cookies and im still using the same ones from last time. in addition the problem still arises with other episodes from this series and others its just 4 (out of 7) of the ones on this series are working for me now

Similar issues here. Happened out of nowhere. Refreshing cookies doesn't matter.

Trying to show my brother how to use mpv+yt-dlp to watch funimation. I can do yt-dlp commands against funi successfully, but when he runs the exact same commands he gets this result:
[funimation:page] kaguya-sama-love-is-war_miko-iino-wants-to-be-soothed-kaguya-doesnt-realize-chika-fujiwara-wants-to-battle: Downloading JSON metadata [Funimation] 4184947: Downloading player webpage for 4184947 [Funimation] miko-iino-wants-to-be-soothed-kaguya-doesnt-realize-chika-fujiwara-wants-to-battle: Downloading Simulcast japanese (4167943) JSON WARNING: [Funimation] Funimation said: Error 4 - No valid Free Avails for this experience [Funimation] miko-iino-wants-to-be-soothed-kaguya-doesnt-realize-chika-fujiwara-wants-to-battle: Downloading Simulcast english (4184947) JSON WARNING: [Funimation] Funimation said: Error 4 - No valid Free Avails for this experience ERROR: [Funimation] 4167941: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

You have to pass cookies to watch such episodes (as it says Error 4 - No valid Free Avails for this experience).

@gw711 You have to pass cookies to watch such episodes (as it says Error 4 - No valid Free Avails for this experience).

But I can do it successfully without cookies, as this episode doesn't even require a login:
`C:\Users\gw>yt-dlp -F https://www.funimation.com/v/kaguya-sama-love-is-war/ai-hayasaka-wants-to-stave-them-off-the-student-council-has-not-achieved-nirvana-kaguya-wants-to-get-
married-kaguya-wants-to-celebrate
[funimation:page] Checking geo-location
[funimation:page] kaguya-sama-love-is-war_ai-hayasaka-wants-to-stave-them-off-the-student-council-has-not-achieved-nirvana-kaguya-wants-to-get-married-kaguya-wants-to-celebrate: Do
wnloading JSON metadata
[Funimation] 1136851: Downloading player webpage for 1136851
[Funimation] ai-hayasaka-wants-to-stave-them-off-the-student-council-has-not-achieved-nirvana-kaguya-wants-to-get-married-kaguya-wants-to-celebrate: Downloading Simulcast japanese
(1079007) JSON
[Funimation] ai-hayasaka-wants-to-stave-them-off-the-student-council-has-not-achieved-nirvana-kaguya-wants-to-get-married-kaguya-wants-to-celebrate: Downloading Simulcast japanese
(1079007) m3u8 information
[Funimation] ai-hayasaka-wants-to-stave-them-off-the-student-council-has-not-achieved-nirvana-kaguya-wants-to-get-married-kaguya-wants-to-celebrate: Downloading Simulcast english (
1136851) JSON
[Funimation] ai-hayasaka-wants-to-stave-them-off-the-student-council-has-not-achieved-nirvana-kaguya-wants-to-get-married-kaguya-wants-to-celebrate: Downloading Simulcast english (
1136851) m3u8 information
[info] Available formats for 1079003:
ID                                  EXT RESOLUTION FPS |   FILESIZE   TBR PROTO  | VCODEC        VBR ACODEC     MORE INFO
1079007-hls-audio-aacl-256-Japanese mp4 audio only     |                  m3u8_n | audio only        unknown    [japanese] Simulcast
1136851-hls-audio-aacl-256-English  mp4 audio only     |                  m3u8_n | audio only        unknown    [english] Simulcast
1079007-mp4                         mp4 unknown        |                  https  | unknown           unknown    [japanese] Simulcast
1136851-mp4                         mp4 unknown        |                  https  | unknown           unknown    [english] Simulcast
1079007-hls-415                     mp4 416x234     23 | ~ 76.60MiB  415k m3u8_n | avc1.4D4028  415k video only [japanese] Simulcast
1136851-hls-416                     mp4 416x234     23 | ~ 76.78MiB  416k m3u8_n | avc1.4D4028  416k video only [english] Simulcast
1079007-hls-574                     mp4 640x360     23 | ~105.94MiB  574k m3u8_n | avc1.4D4028  574k video only [japanese] Simulcast
1136851-hls-577                     mp4 640x360     23 | ~106.50MiB  577k m3u8_n | avc1.4D4028  577k video only [english] Simulcast
1079007-hls-848                     mp4 768x432     23 | ~156.52MiB  848k m3u8_n | avc1.4D4028  848k video only [japanese] Simulcast
1136851-hls-848                     mp4 768x432     23 | ~156.52MiB  848k m3u8_n | avc1.4D4028  848k video only [english] Simulcast
1136851-hls-1146                    mp4 768x432     23 | ~211.52MiB 1146k m3u8_n | avc1.4D4028 1146k video only [english] Simulcast
1079007-hls-1148                    mp4 768x432     23 | ~211.89MiB 1148k m3u8_n | avc1.4D4028 1148k video only [japanese] Simulcast
1079007-hls-1810                    mp4 960x540     23 | ~334.07MiB 1810k m3u8_n | avc1.4D4028 1810k video only [japanese] Simulcast
1136851-hls-1811                    mp4 960x540     23 | ~334.26MiB 1811k m3u8_n | avc1.4D4028 1811k video only [english] Simulcast
1136851-hls-2497                    mp4 1280x720    23 | ~460.87MiB 2497k m3u8_n | avc1.640028 2497k video only [english] Simulcast
1079007-hls-2506                    mp4 1280x720    23 | ~462.53MiB 2506k m3u8_n | avc1.640028 2506k video only [japanese] Simulcast
1136851-hls-3722                    mp4 1280x720    23 | ~686.97MiB 3722k m3u8_n | avc1.640028 3722k video only [english] Simulcast
1079007-hls-3742                    mp4 1280x720    23 | ~690.66MiB 3742k m3u8_n | avc1.640028 3742k video only [japanese] Simulcast
1136851-hls-4834                    mp4 1920x1080   23 | ~892.21MiB 4834k m3u8_n | avc1.640028 4834k video only [english] Simulcast
1079007-hls-4955                    mp4 1920x1080   23 | ~914.55MiB 4955k m3u8_n | avc1.640028 4955k video only [japanese] Simulcast
1136851-hls-6365                    mp4 1920x1080   23 | ~  1.15GiB 6365k m3u8_n | avc1.640028 6365k video only [english] Simulcast
1079007-hls-6523                    mp4 1920x1080   23 | ~  1.18GiB 6523k m3u8_n | avc1.640028 6523k video only [japanese] Simulcast
C:\Users\gw>`
I have a config that passes --no-check-certificate to get around another issue, but anyway I don't need cookies for this. It's just my brother that gets the Error 4 issue.

@gw711 can we see the full command for starters?


Current interceptor chains are not compatible with Axios interceptors

See: https://github.com/axios/axios?tab=readme-ov-file#multiple-interceptors
Currently with xior - if the response interceptor throws, the chain ends: https://github.com/suhaotian/xior/blob/main/src/xior.ts#L239
I think it would be ideal to follow the same logic that axios uses to aid with drop-in replacements. In the case of the xior-auth-refresh plugin, I've noticed that any subsequent response error interceptors won't run.

I see, will fix in next version Thanks feedback!!   🙌

Merged then the issue automatic closed! (well done, github!)

Great thanks so much for this!

Fully validated, so pushed an updated version of the auth plugin which makes sure this version is used so avoid any unexpected behaviour!
Once again, thanks so much for the incredible turnaround - this library has been a lifesaver!

Cool, Thanks! @mgoodfellow 🙌


on_save_checkpoint runs multiple times on DDP

🐛 Bug
on_save_checkpoint handler of the LightningModule runs once per GPU when running on multiple GPUs with strategy=dpp. This can have unwanted side effects when saving additional checkpoint information.
Is there a way to make this run only the master?
Is this the expected behavior? I would think the former would be the more common use case.
To Reproduce
from transformers import AutoModel
from pytorch_lightning import LightningModule

class BoringModel(LightningModule):
    def __init__(self,  model_name, save_path):
        super().__init__()
        self.save_hyperparameters()
        self.model = AutoModel.from_pretained(self.hparams.model_name)

    def forward(self, pixel_values):
        outputs = self.model(pixel_values=pixel_values)
        return outputs

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)

    def on_save_checkpoint(self, checkpoint):
        print("Saving model latest checkpoint in HF format..")
        self.model.save_pretrained(self.hparams.save_path)

Lightning Component (e.g. Trainer, LightningModule):
PyTorch Lightning Version  1.6.4
OS (e.g., Linux): Linux
CUDA/cuDNN version: 11.3
GPU models and configuration:  8xT4
Any other relevant information:

Is this the expected behavior?

Yes, it definitely is.

This can have unwanted side effects when saving additional checkpoint information.

What for example? Can you give an example?
on_save_checkpoint's primary use is to modify the dictionary checkpoint before it gets saved. Whether all ranks do something with this dictionary or not is up to the implementation in the strategy. Example: Sharded models may want to save a portion of the weights on each rank.

Is there a way to make this run only the master?

def on_save_checkpoint(self, checkpoint):
        if self.global_rank == 0:
            # run only on rank 0

Closing the issue as behavior is expected. In general, all hooks need to run on all ranks.
If you are looking to save only on one rank, this can be accounted for by:
def on_save_checkpoint(self, checkpoint):
        print("Saving model latest checkpoint in HF format..")
        if self.global_rank == 0:
            self.model.save_pretrained(self.hparams.save_path)

@awaelchli  Thanks for the work around.


Can't send email

Win 10, Chrome 74.0.3729.169 (Official Build) (64-bit)
Steps:

Click button Test Runs
Click Burger button on bottom-right corner
Click button "send as email"
Enter your email
Click button "Send"
Actual result: Nothing happens
Expected result: Email sent

Not repro

Today this button is absent

@ipapelishvili , the bug is not repro, could you please provide us url to the app, video with steps to reproduce

Not repro


Merge develop branch to main branch and create tag v1.0.0_impl , v1.0.0_test

Merging develop to main branch

[x] Create pull request to merge develop branch to main branch
[x] Create a tag v1.0.0_impl with the following details of the open issues in the version 1.
[x] Create a tag v1.0.0_test for publishing the testsuites for version 1.

Implementation of TypeApprovalRegister v1.0.0
This release contains,

implementation of the individual services
BasicServices incorporated from npm package onf-core-model-ap-bs+1.0.1
Application pattern framework incorporated from npm package onf-core-model-ap+1.0.1

Open Issues
Report timestamp : Wednesday, 23 November 2022 14:57:51
Total run Duration : 10m 44.7s
Average response time : 57ms
Total assertions : 1694
Total failures : 65
Total skipped test : 0
Consistent Issues :

Categories
count
Rootcause
Issue number / further reference

AssertionError - Expected /v1/bequeath-your-data-and-die - incorrect new-application-name attribute
1
For incorrect application-name , server is not processing the request. This behavior is fine. But the response code should be 400 instead of 204.
TypeApprovalRegister/issues/76

AssertionError - BequeathingDataAndDieCausesForwardings - reference OldRelease/core-model-1-4:control-construct
1
OldRelease IP not configured in the application and resulting in request timeout. Issue will be resolved after including "old-release" information in the embed-yourself callback in registry-office.
RegistryOffice/issues/108

RangeError - BequeathingDataAndDieCausesForwardings - sampling /v1/bequeath-your-data-and-die
1
OldRelease IP not configured in the application and resulting in request timeout. Issue will be resolved after including "old-release" information in the embed-yourself callback in registry-office.
RegistryOffice/issues/108

AssertionError - BequeathingDataAndDieCausesForwardings - sampling /v1/bequeath-your-data-and-die
1
The application request body received is with "undefined" values. Server is not processing the request.
Issue will not be observed after updating testsuite to version 2

AssertionError - Response body completeness - Expected /v1/start-application-in-generic-representation
4
The datatype "String" changed from capital "S" to small "s" in the version 2 specification. This change is implemented in the server side. And in the version 2 of testsuite this will be included.
Issue will not be observed after updating testsuite to version 2

AssertionError - Response body completeness - Expected /v1/list-applications
1
As per the test result , the ControlConstruct and the number of application is 9. This shall be not observed after updating the testsuite to version 2

AssertionError - Response body completeness - Expected /v1/list-approved-applications-in-generic-representation
8
IThe datatype "String" changed from capital "S" to small "s" in the version 2 specification. This change is implemented in the server side. And in the version 2 of testsuite this will be included.
Issue will not be observed after updating testsuite to version 2

AssertionError - Response body completeness - Expected /core-model-1-4:control-construct
11
This consists of 2 categories of issues , 6 - mismatch in the IP address between load file in testsuite and server. Part of fake IP address this will be fixed. 5 - new operation-client creation updating new operations without the proper versioning creates new LTPs instead of updating existing.
mismatch in the IP address - will not be observed from the next release of testsuite , new operation-client creation - ApplicationPattern/issues/284

Inconsistent Issues :
The following issues are inconsistent and will be fixed after supporting ElasticSearch in the EaTL and OL application ApplicationPattern/issues/357,

Categories
count

ExecutionAndTraceLog/core-model-1-4:control-construct
15

Expected ExecutionAndTraceLog/v1/list-records-of-flow
17

Expected OamLog/v1/list-records-of-application
5

Merged the changes to main branch and created tags v1.0.0_impl and v1.0.0_test. Hence closing this issue.


Code Quality: Add an unit test for Files.App

Description
My long-long-term plans have been to remove Utils namespace and unify with injectable Services via Ioc and thus to create unit test in terms of managed codebase.
The last try to create WinUI unit test was failed for some reasons.
When I referenced 'Files.Package' project it said unrecognizable platform identifiers, when I referenced 'Files.App' project it said it's not targetable iirc.
@marcelwgn I saw your Visual Studio template for WinUI unit tests, do you happen to know how to reference Files.App correctly and execute tests?
Concerned code

Whole Files.App

Gains

Better maintainable experience for us

Requirements

Have tests for Files.App.Services
Have tests for some controls in UserControls such as ThemedIcon (we have an ongoing PR and it holds test window but it's not ideal at all.

Comments
No response

Yeah referencing a WinUI 3 app from another WinUI 3 app is not possible (or I don't know what would help). It seems we are not the only people with that issue: https://github.com/microsoft/microsoft-ui-xaml/issues/6279#issuecomment-1745324868

Thank you for the info. I’ll close as it requires WinUI effort…


This method does not work on the current Raspbian version

Please update it for Stretch, as this thing is broken on stretch.
If you run it, you'll run into dependency issues, as this thing uses pip2. Replacing pip2 with pip3, does finish the installation, but then the web_auth script doesn't work. Also replacing python2 with python3, gives garbled URLS in the authentication part, which don't work (lots of %20 and so forth in there, and in general the URLS are wrong (ie + "code", why?).
This method was being advertised as easier than using the official Alexa repo's, but it cost me about one and a half day of tinkering, and then still didn't work.
Expected result:

Working Installation
God Result:
Not working Installation

To reproduce, just get a current raspbian stretch image and follow the installation instructions. It's broken.

It took me a few hours tonight but I finally got it working, using the latest Raspbian.
I to ran into issues with python / pip / blah
then having to use
sudo pyhton3 auth_web.py
to get the auth_web to work properly
then it was a matter of going through the Pulse Audio stuff and modify the config.yaml
to match up to that stuff and get the usb microphone in the input settings
and finally it works.

Same here, it is broken due to python and pip versions, still no luck fixing it


New Quest - HM01 Cut (Lance + SSAnne + All Items)

Tested with secondary account and work perfectly :)

Remember to also add the quests in https://github.com/g0ldPRO/Questing.lua/blob/master/Quests/QuestManager.lua ;)

As already discussed with Rympex, SSAnne should be a quest on its own since being on SSAnne map will always mean we are doing the SSAnne quest.

As already discussed with Rympex, SSAnne should be a quest on its own since being on SSAnne map will always mean we are doing the SSAnne quest.

:+1: working nicely.
Talking twice to the NPC, but not a big deal.

Edit done i will post the new quest


Support Prepared Statements

Expected Behavior
Nanodbc supports prepared statements. The same should be exposed by Cyanodbc in a DB_API compatible manner.
Specifications

Version: 0.0.1
Platform: All
Subsystem:

PR#12 will add this missing functionality


Issue credits to Omo

Issue 12000 credits to @Omoaholo.

 Follow up
On Wednesday, 10-Apr-24 17:09:31 UTC by Gov4Git dev
Issued 12000 credits to member @omoaholo.


NSM + Istio booking example doesn't work

Expected Behavior
NSM + Istio booking example is working correctly.
Current Behavior
NSM + Istio booking example doesn't workg.
Steps to Reproduce
Run https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/interdomain/nsm_istio_booking
Context
NSM Version: v1.6.1
Failure Logs
fail_response.txt
ok_response.txt

Still reproducaible with NSM v1.8.0

Fixed by https://github.com/networkservicemesh/sdk/pull/1443


Enable more dynamic source attribute

Certainly the idea of versioning the modules in a separate git repo is great. But while developing it quickly becomes a hassle to specify a different source for every module. Also, the support for specifying a source is not really fully complete, as the --terragrunt-source argument "is not used with the apply-all, destroy-all, output-all and plan-all commands", effectively preventing testing of these vital commands during development.
Would it be a good idea to add an argument like --terragrunt-source-prefix or --terragrunt-source-root, with corresponding environment variable, so that users can set the source once during the development process, and not have to change it all the time?

A --terragrunt-source-root command line param and corresponding env var (TERRAGRUNT_SOURCE_ROOT) sounds like a great idea! PR very welcome :)

Would the thinking be that you would replace everything before the // which indicates a subdirectory of a git repo?


Time Ontology -small suggestions

I've been comparing the Time Ontology to the OWL Time Ontology. Here are some small suggestions for the CCO:

There are a few relations in CCO-Time that seem like sub relations of another relation. For instance, 'interval is after' and 'interval is before' are both sub relations of 'interval disjoint'.

I don't see 'Time Zones' represented in the CCO's Time. Is there another way they are being treated?

Is there a place in the CCO for temporal reference systems?

For consistency, why not treat the subclasses of one-dimensional temporal region presently in the CCO (e.g. Day, Decade, Hour, Minute, etc) as instances of temporal units like the measurements of units ontology does for units?

This is interesting. I think you're correct that before/after are subpropeties of disjoint, considering disjoint is defined in terms of one interval that is "before OR meets" the other. Before/after properties are about intervals which meet, therefore I don't see why they wouldn't qualify as subproperties.

At present, time zones are represented in terms of an identifier attached to the IBE which has the datetime information. Specifically, the class "Time Zone Identifier" is in our Information Entity Ontology, and is annotated in part as follows:

"There is no class 'Time Zone' (the region), only 'Time Zone Identifier' (the designator for some region). Instances of 'Time Zone Identifier' should be related to instances of 'Information Bearing Entity' by means of the object properites 'uses time zone identifier' and 'time zone identifier used by' as appropriate..."
The reason is basically that it would be difficult, if not impossible, to represent the relationship between data which references a timezone and the timezone-qua-region. I might have a piece of data which timestamps something as happening at 12:00 GMT, but what's the relationship between the geospatial region delineated as GMT and my data? I can't, for example, presume that the event I'm describing occurs somewhere in that region.

Information Entity Ontology also includes "Reference System," including the subclass "Temporal Reference System," which has several subclasses. Again, such reference systems are related to our primary temporal data via the IBE. E.g., our data reads "2018-05-21T12:00:00" and we append to the IBE the fact that the "Solar Calendar System" being used here is the instance "Gregorian Calendar".

Because "Day" is not an instance, but a class, of which May 20 and May 21 are instances.

P.S. If the "Time Ontology in OWL" isn't called "TOWL," then somebody needs to ask for their money back.

Thanks for the helpful explanations. You might also see in 'TOWL' the relation 'interval in' whose subs include interval during, interval finishes, and interval starts.
Concerning your comment on 'Day', the same could be said of 'inches'. For instance, there is the instance of a measurement unit of length 'inch', but then there's this inch, which is a one-dimensional spatial extent that is measured by the instance of measurement of length 'inch'. Since you don't need to represent both inch-qua-informational-entity and inch-qua-unitized-quantity-kind in your ontology, you only need choose one way of going. And the CCO does exactly this. I was just pointing out that it appears one methodology was used for units of measurement (the former), while the second was used in the Time ontology.

There's an ambiguity between designating a day-long 1d temporal interval and measuring a region in units of 1 or more days. Yes, the former is based in some way on the latter. But I wouldn't say there's a redundancy here, for the simple reason that people are in the habit of designating temporal regions (e.g., "May 21, 2018), but not of measuring the interval from 0 C.E. to present using measuring units.

The object properties interval_meets, interval_met_by, interval_is_before, and interval_is_after are now all sub-properties of interval_disjoint.


feat: add coverage support

~Note that this doesn't really work - the commented out filter needs to be there in some way.~ I think it's ready
However, copying this over into prettier seems to work fine.

Fixes #6.

@nicolo-ribaudo I have zero attachment to this particular approach, feel free to slice and dice it as needed 🙂 it seems to work fine, at least

I'm not near my laptop, will test later. Good job!

I got an error, I don't know what is it... https://github.com/prettier/prettier/runs/5861202752?check_suite_focus=true#step:8:14

Because Node.js 12 don't support private field?

wow, what a useless error message 😅

@fisker pushed again, works on node 12 on my machine

All good, got the coverage https://github.com/prettier/prettier/runs/5863566812?check_suite_focus=true#step:6:1139
Thanks again!

awesome! it looks like expected coming from c8?

c8 says 96.5% https://github.com/prettier/prettier/runs/5850917095?check_suite_focus=true#step:6:1767
this one says 72.4% https://github.com/prettier/prettier/runs/5863566812?check_suite_focus=true#step:6:1142

oh, haha

Right, seems to be missing coverage of the CLI. You probably spawn it instead of executing from within a test?

Our next version not running CLI test, remember this https://github.com/prettier/prettier/pull/12271 ?

Aha! 😀 And the light runner doesn't solve the issue?

We are using different runners, only format test use this runner https://github.com/prettier/prettier/blob/001f8fb086eefb73abefa46eba20370d5f0e84f0/jest-format-test.config.mjs#L5

This is not important, but just a question, why this collectCoverage makes Jest much slower than running with c8?

I'm guessing because c8 only hooks up the inspector once instead of per test

It looks like this breaks Babel's coverage collection, making it go down from ~90% to 0%. I just added coverageProvider: "v8" to https://github.com/babel/babel/blob/main/jest.config.js and run make test-ci-coverage.

6dfa6f970c2d9e6b7835ba903ce16804d2f39f51 is the minimum to "unbreak" Babel's tests. Feel free to revert it if you know what is the root cause / proper fix :sweat_smile:

@nicolo-ribaudo seems fine 🙂 But coverage for the babel repo is broken using v8 coverage? sounds like a bug I need to fix 😅

Could it be because we test the compiled files, but want coverage in the original sources?

as long as there's a sourcemap that shouldn't be an issue. Jest has some custom mapping, maybe something like that is needed? https://github.com/facebook/jest/blob/a5f1ef43981b7426d61bcd7cbc59a79f548c075a/scripts/mapCoverage.mjs

But Prettier use the source directly, the coverage still lower.

Some smaller reproduction than prettier or babel would be nice to investigate 🙂 I think this PR can still land in the meantime.
One interesting check would be to check coverage using v8 provider with the default jest runner and if it differs from the one from the light runner

I have simple tests in my repo. You can check it out: https://github.com/psychobolt/yeoman-generator-boilerplate/pull/20

Rebased this FWIW. I haven't had the time to dig into why it's different

c8 says 96.5% prettier/prettier/runs/5850917095?check_suite_focus=true#step:6:1767 this one says 72.4% prettier/prettier/runs/5863566812?check_suite_focus=true#step:6:1142

I have a possibly related issue, when I reset globalThis.__coverage__ before each test run, babel's coverage drops by 30%.
globalThis.__coverage__ = undefined
It seems that globalThis.__coverage__ is not being merged properly.
https://github.com/nicolo-ribaudo/jest-light-runner/blob/83da92ae0c2f79152aa50af6baa286f3583fb753/src/worker-runner.js#L246

I debugged in babel.
Just removing shouldInstrument() produces coverage output (although coverage is currently around 10% less), since babel is precompiled and does not use jest-transform.
But this also makes coveragePathIgnorePatterns completely invalid, maybe we can move shouldInstrument() to after restoring the source file path using sourcemap?
I also found a problem.
babel-helper-transform-fixture-test-runner will run the dummy file on the vm, which will cause the jest _getCoverageResult method to throw an exception because the file cannot be read.
https://github.com/babel/babel/blob/53e9057d3022cdf41c2593261b8f5c4a4e26265d/packages/babel-helper-transform-fixture-test-runner/src/index.ts#L99-L107

That last one sounds like a bug in jest - would you mind opening an issue (in jest)? Reproduction of babel repo is fine

Unfortunately cannot reproduce in jest-runtime because dummy files are not in this._v8CoverageSources.😰
https://github.com/facebook/jest/blob/561907ce5d1e9065311cf585f75b9c5864036193/packages/jest-runtime/src/index.ts#L1181-L1190

Huh. So it's something in this runner

Yes, it seems to be a compatibility issue.
I just applied some skips manually and now v8 and babel have the same result on my local (windows)!
I'll try to parse the source map next to accomplish this filtering.

What do you mean with "doesn't support"? Is the pattern not supported? Not respected? Sounds like a bug still, but somewhat hard to nail down 😅

That is, the source code is in ./src, and the compiled file is in ./lib.
collectCoverageFrom must be set to ./lib to work.
I think this is acceptable behavior whether it's a bug or not.
Also I seem to find a real bug, when I enable a different number of coverageReporters, the strange thing is that the coverage shown changes, it seems that some reporters are changing the original coverage data.

It seems to be closer to the real cause.
When I use maxWorkers=2, the result is 80%. When I use maxWorkers=16, the result is 90%.
This might also explain https://github.com/nicolo-ribaudo/jest-light-runner/pull/13#issuecomment-1242827537 and https://github.com/nicolo-ribaudo/jest-light-runner/pull/13#issuecomment-1091155503

Lifted over these changes into my own fork of (jest-light-runner)[https://github.com/Alx101/jest-light-runner/tree/coverage-and-globals] (along with some experiments for global setups before all test-suites, still not working). Since I'm running this on a TS project, i've noticed that the coverage report line numbers seem incorrect, probably it's not converting with the source-map?


screen becomes blurred and indistinct in the Gamescope

Describe the bug
After reboot the computer I meet the problem that my screen becomes blurred and indistinct like the below picture.But if I move my mouse,the screen becomes normal. However when I use my controller to switch,the screen still like this.
And sometimes when I switch to desktop mode,the screen is white, with chromatic aberrations.

What did you expect to happen?
I want this problem will be sloved.
Output of rpm-ostree status
State: busy
Transaction: upgrade
  Initiator: client(id:cli dbus:1.227 unit:run-r6a798f51b6df47e8a5d2058559402c45.scope uid:1000)
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-deck:stable
                   Digest: sha256:82624ab8dc34f83515f74cdaa160683f342cc62b8dd2d485fa2d465385ec5fab
                  Version: 40.20240717.0 (2024-07-17T17:10:50Z)
          LayeredPackages: nss-tools

  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-deck:stable
                   Digest: sha256:9dd5cf961270659215f5a20a2884eef5ce4d7c203bac4de1125cbc32aa34c6fe
                  Version: 40.20240716.0 (2024-07-16T04:58:03Z)
          LayeredPackages: nss-tools

Hardware
AMD R7-5700X
AMD RX7700XT
B550I
32G RAM
Extra information or context
No response

This is an issue with HDR in GameScope, I would make sure that HDR is enabled both in your monitor and in GameScope. If you still can't get it to work, try making sure both are off.


CI: golangci-lint fails locally

The golangci-lint target fails when I run it locally:
$ make golangci-lint
docker run --rm -v `pwd`:/app -w /app docker.io/golangci/golangci-lint:v1.54.0@sha256:fcf89904b0a852690374e1624438837e438ba8518a12b0e7684683758185b7ff golangci-lint run
bugtool/main.go:1: : error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping. (typecheck)
// SPDX-License-Identifier: Apache-2.0
cilium/main.go:1: : error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping. (typecheck)
// SPDX-License-Identifier: Apache-2.0
cilium-health/main.go:1: : error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping. (typecheck)
// SPDX-License-Identifier: Apache-2.0
make: *** [Makefile:421: golangci-lint] Error 1

Additional information:
$ go version
go version go1.21.0 linux/arm64

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

xref: https://github.com/cilium/cilium/issues/25155

I've just tried this now and I can't reproduce, similar but slightly newer Ubuntu LTS. I see that the golangci-lint image has been bumped since the version listed in your output, maybe this was already resolved?


BE | Implement language selection

The simplest way to implement language selections/activation on the BE by the FE (client) would be to pass it as a URL parameter.
Implement a middleware that will take (pop) that language from the URL query parameters and activate it so that the specified language is active during the entire request lifecycle.
This will make sure that all translatable fields (e.g. FAQ questions and answers are returned in the specified language.

Implemented (last commit)