HDDS-9224. Recon HeatMap UI issues

What changes were proposed in this pull request?
Few HeatMap issues:

Not able to enter "/s3v/cloudera-health-monitoring-ozone-basic-canary-bucket/cloudera-health-monitoring-ozone-basic-canary-key" in path box.
Tooltip always show FileName label irrespective of entity type selected. It should display label as Key Name, if selected bucket entity type, tool tip should have label Bucket Name and if volume, the should be Volume Name.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9224
How was this patch tested?
Manually

I'm no too familiar with this, but based on the screenshots for the keys you are showing Key Name, which seems correct for me. on the 2nd screenshot, when the cursor is on a bucket, the bucket's name is test11.dat? or is it showing a wrong name? also hivebucketTest2 is a volume? what would happen if the cursor is on the HIVEVOLTEST2 and what is that? can you attach a basic example of vol1, bucket1 and key1? or just add the names and types of the entities in a comment?

@Zita
Earlier in tooltip FileName label  was static for key volume and bucket so as per suggestion we changed to dynamic if it is Key we are replacing Key Name, if it is volume we are changing to Volume Name if it is bucket we are chnaging to Bucket Name.

I'm no too familiar with this, but based on the screenshots for the keys you are showing Key Name, which seems correct for me. on the 2nd screenshot, when the cursor is on a bucket, the bucket's name is test11.dat? or is it showing a wrong name? also hivebucketTest2 is a volume? what would happen if the cursor is on the HIVEVOLTEST2 and what is that? can you attach a basic example of vol1, bucket1 and key1? or just add the names and types of the entities in a comment?

@zita Earlier in tooltip FileName label was static for key volume and bucket so as per suggestion we changed to dynamic if it is Key we are replacing Key Name, if it is volume we are changing to Volume Name if it is bucket we are chnaging to Bucket Name.

I understand the changes, but I'm not sure that the current implementation is correct. To make it sure, I'd like to know the types of the entities that we are seeing in the screenshots. Currently the volume/bucket/key names seems to be not so straightforward, e.g.:

Volume Name: hivebucketTest2 (it contains the bucket word)
Bucket Name: test11.dat (this seems to be a file, based on the extension)

I have a detailed list of the names/types in my previous comment.
Could you upload screenshots with simple vol1, bucket1 and key1 names?

m not sure that the current implementation is correct. To

I'm no too familiar with this, but based on the screenshots for the keys you are showing Key Name, which seems correct for me. on the 2nd screenshot, when the cursor is on a bucket, the bucket's name is test11.dat? or is it showing a wrong name? also hivebucketTest2 is a volume? what would happen if the cursor is on the HIVEVOLTEST2 and what is that? can you attach a basic example of vol1, bucket1 and key1? or just add the names and types of the entities in a comment?

@zita Earlier in tooltip FileName label was static for key volume and bucket so as per suggestion we changed to dynamic if it is Key we are replacing Key Name, if it is volume we are changing to Volume Name if it is bucket we are chnaging to Bucket Name.

I understand the changes, but I'm not sure that the current implementation is correct. To make it sure, I'd like to know the types of the entities that we are seeing in the screenshots. Currently the volume/bucket/key names seems to be not so straightforward, e.g. (these are copied from the screenshot):

Volume Name: hivebucketTest2 (it contains the bucket word)
Bucket Name: test11.dat (this seems to be a file, based on the extension)

I have a detailed list of the names/types in my previous comment. Could you upload screenshots with simple vol1, bucket1 and key1 names?

Can you ignore data as is mockdata for local check. I have to change Bucket and Volume and Keys DB.json ata as per cluster.data.

@smitajoshi12 I can't ignore data, as I don't have too much else to test the functionality of your change. Also if you attach this as testing, it should show correct data. I'm still not sure if the implementation is correct.
I think the most outstanding thing is that on the last screenshot the tooltip is coming from the bucket, which has the name HIVEBUCKETTEST2 based on the heatmap figure. This tooltip shows Volume Name: and after that the bucket name is showing up. This is definitely wrong. Either it should show the volume name (which is the "parent") or it should be a "Bucket Name". This can be figured out even if the names are "incorrect" to their type.
Also I don't think that the childEntityType in the code is the correct name, as we are talking about the parents of the entites (eg. for a bucket, it has a "parent" volume; it can have more child entities, so it wouldn't make sense to show that).
I wanted to try this out in a docker cluster locally with my data, but I'm not sure how can I enable heatmap, I don't see it there default? Can you help me with this? @devmadhuu

@smitajoshi12 I can't ignore data, as I don't have too much else to test the functionality of your change. Also if you attach this as testing, it should show correct data. I'm still not sure if the implementation is correct.
I think the most outstanding thing is that on the last screenshot the tooltip is coming from the bucket, which has the name HIVEBUCKETTEST2 based on the heatmap figure. This tooltip shows Volume Name: and after that the bucket name is showing up. This is definitely wrong. Either it should show the volume name (which is the "parent") or it should be a "Bucket Name". This can be figured out even if the names are "incorrect" to their type. Can you look into this a bit more? Thanks!
Also I don't think that the childEntityType in the code is the correct name, as we are talking about the parents of the entites (eg. for a bucket, it has a "parent" volume; it can have more child entities, so it wouldn't make sense to show that).
I wanted to try this out in a docker cluster locally with my data, but I'm not sure how can I enable heatmap, I don't see it there default? Can you help me with this @devmadhuu? Thanks!

@smitajoshi12 , you can show screenshots with your db.json data in each case (volume, bucket, key, file)...
@dombizita , in upstream this will not be enabled , so cannot test with live data in upstream, however yes, you can create an internal cluster to test.

could take a look at the latest patch @devmadhuu, @devabhishekpal?

thanks for the patch @smitajoshi12! thanks for the review @devabhishekpal!


SerialPort with Pinpad Device

Summary of Problem
I developed an EFT / TEF application on windows 10 using a USB pinpad, however the communication is based on the exchange of messages between my application and an application on the pinpad known as a shared library, which has the EMV commands for use in the card.
I made a small test program that sends and waits for the response to three commands: Closing, Opening and Discovering pinpad details, manufacturer, serial number, firmware version, etc.
but I have two approaches and two problems
to test scenario (1) set MODOFECHA=1, another scenario (2) set MODOFECHA = 0
(Please answer all 3)

What are you trying to do?
the application starts by sending a logical close command with the shared library and waits for response, then sends the opening command with the shared library waiting for response and then the command to discover details of the pinpad is also waiting for response and will do this for 500 iterations

What happens? 2 cenarios same source to test

when closing the serial port and cleaning the port controller (port = null)
I have problem opening the serial on the i5 / core client machine, but this does not happen on my i7 / core machine

when I close the serial port and do not clean the port controller, I have a problem with port.on (date) and with each port.write, the higher the number of shots occur

What should have happened?

open succesfully
a single receipt shot

Code to Reproduce the Issue
see inside zip file
Versions, Operating System and Hardware
6 May 17:45:40:522 - [info] Electron Version:2.0.9
6 May 17:45:40:538 - [info] Platform:win32
6 May 17:45:40:538 - [info] CPUs:Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz - 2904Mhz

SerialPort@?  7.1.5 extract from package.json
Node.js v? -- v8.15.0
Windows? Linux? Mac? win32    -- App(Win32)/(Machine)Windows10x64
Hardware and chipset? (Prolific/FTDI/Other)

Suporte.zip
Biblioteca compartilhada - v108a.pdf

Solution: Destroying the Event Emitter Listener  [   port.removeListener('data',.....)   ],
However the error in opening the communication port the first time persists, after some error is opened


Beta Example Tests

This PR is runs example tests of beta features if the feature flag enable-api-fields is set to beta. This is an extension to PR https://github.com/tektoncd/pipeline/pull/5737. It also addresses a part of https://github.com/tektoncd/pipeline/issues/5734

Changes

Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[x] Has Docs included if any changes are user facing
[x] Has Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
[x] Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
[ ] Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes
Run beta example tests when `enable-api-fields: beta` is set.

/kind feature

@chitrangpatel @JeromeJu does this address https://github.com/tektoncd/pipeline/issues/5734?

There has not been yet any examples tests for now but I wonder if it might be helpful to include addition of paths in https://github.com/tektoncd/pipeline/blob/main/test/examples_test.go#L248-L269 as for the purpose/title of this PR?

/assign

Thanks @chitrangpatel for setting up the examples tests for beta:)
I am aware that there has not yet been any examples tests but I wonder if it might be helpful to include addition of paths in https://github.com/tektoncd/pipeline/blob/main/test/examples_test.go#L248-L269 as for the purpose/title of this PR?

Hmm. I'm not sure anything needs to be updated there. What were you thinking of in particular?
The function you referenced is called by TestExamples which does the path filtering before calling it.

/retest

https://github.com/tektoncd/pipeline/blob/main/test/examples_test.go#L248-L269

Right! sorry I was thinking ahead of when, in the future, we deprecate v1beta1 but for now there should be nothing to change maybe we can have a similar TODO with https://github.com/tektoncd/pipeline/issues/5865 for waitValidatePipelineRunDone and waitValidateTaskRunDone here 😺

/lgtm

/hold

/lgtm

/hold

@lbernick can we cancel the hold now?

/hold cancel


textures isssues

Yes, this is a temporary measure. at the moment, I can't add textures to them since there are no sources in this style.

will it be fixed in future ?

yes it will be fixed.


[SR-14173] Referencing operator function '!=' on 'BinaryInteger' requires that enum with associated value conform to 'BinaryInteger'

Previous ID
SR-14173

Radar
rdar://problem/74149099

Original Reporter
@idrougge

Type
Bug

Environment
Version 12.3 (12C33)

Additional Detail from JIRA

Votes
0

Component/s
Compiler

Labels
Bug, DiagnosticsQoI, TypeChecker

Assignee
None

Priority
Medium

md5: e6ecba6fdfe08bae56e4fc622a8187eb

Issue Description:
Given an enum, where one case has an associated value:
enum E {
 case a
 case b(Double)
 case c
}

When doing a comparison of a value v: E, as below:
if v != .c { … }

the compiler puts out this error message:
Referencing operator function '!=' on 'BinaryInteger' requires that 'E' conform to 'BinaryInteger'.
This error message is misleading since at no point is any reference to BinaryInteger or its ![](= made, the underlying error being that one of the enum cases has an associated value and that the )= operator needs to be implemented or synthesised by declaring Comparable conformance.

I get this error
Referencing operator function '!=' on 'BinaryInteger' requires that 'MyEnum' conform to 'BinaryInteger' and no mention of associated values.


Server-side authentication using cookies

I think we need to make this straight.
The current authentication is on client-side which using localStorate
Let's actually finish this up.

Done here! #11


[Service Bus] Peeking into an empty session-enabled queue throws an error

Peeking into an empty session-enabled queue would run into either of the following errors..

OperationTimeoutError: Unable to create the amqp receiver 'unpartitioned-queue-sessions-794f89be-3282-8b48-8ae0-a8af43c3ce36'  on amqp session 'local-1_remote-1_connection-2' due to operation timeout.
MessagingError: Received an incorrect sessionId 'undefined' while creating the receiver 'unpartitioned-queue-sessions-86662b2b-acdc-1045-8ad4-fa3ab8807871'.

Not completely sure if this is due to opening multiple connections or multiple receivers (might be a drawback of the 2-client API). Need to revisit and investigate once the API with Top-level client is finalized and implemented.

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=281562&view=logs&j=a556eced-ff92-56f9-81ae-670d7020e9e7&t=19299f15-c0e5-5217-5db1-bd33a9df4882
Commit to repro - 98563ca33c640fe3e93500da2bbf105373ae0cc0

Am guessing the behavior is now different given that one can peek only after creating the session receiver

Is this for the case of named session or unnamed session?
If this is for Track 1, then is it for the peek() operation on the QueueClient/SubscriptionClient or the peek() operation on the receiver?

Is this for the case of named session or unnamed session?

It was unnamed session.

If this is for Track 1, then is it for the peek() operation on the QueueClient/SubscriptionClient or the peek() operation on the receiver?

Not Track 1, I'd say Track 1.5, while designing track 2. Peek is only on the receiver, there was no QClient/SubClient at this point.

This issue no longer repros with the current API in track 2. Hence, closing.


AC1214/10

I did  all of the lines:
"Update CoAPthon3 using sudo pip3 install -U git+https://github.com/Tanganelli/CoAPthon3@89d5173.
For new firmware version
(Who get error Unexpected error:'NoneType' object has no attribute "payload")
Found coap_client.py path using sudo find / -name *coap_client.py.
Open file coap_client.py and line 91 change timeout from 5 to 60,
line 145 from 5 to 60 and line 174 and timeout=60 after encrypted_payload
Change in plugin settings Timeout Seconds to 30 or 60."
And error still exist:
23/07/2021, 01:02:52Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@@Model] characteristic value expected string and received undefined
23/07/2021, 01:02:52Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@@Firmware Revision] characteristic value expected string and received undefined
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@Purifier@Active] characteristic value expected valid finite number and received "undefined" (undefined)
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@Purifier@Current Air Purifier State] characteristic value expected valid finite number and received "NaN" (number)
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@Purifier@Lock Physical Controls] characteristic value expected valid finite number and received "undefined" (undefined)
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@Air quality@Air Quality] characteristic value expected valid finite number and received "NaN" (number)
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@Air <EMAIL_ADDRESS>Density] characteristic value expected valid finite number and received "undefined" (undefined)
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@Pre-filter@Filter Life Level] characteristic value expected valid finite number and received "NaN" (number)
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@Active carbon filter@Filter Life Level] characteristic value expected valid finite number and received "NaN" (number)
23/07/2021, 01:02:53Philips Air BridgeERROR[Philips Air Bridge 3F0E@Purifier@HEPA filter@Filter Life Level] characteristic value expected valid finite number and received "NaN" (number)
And again after 1-2min, when I turn it off from HomeKit, it shows is ON, status issue.

Having the same happen to me with the Philips AC3033/10. Then again, py-air-control also keeps on coming up with no data every 2 out of 3-4 attempts. It takes multiple times to try to get all stats. The Clean Home+ app doesn't have issues
And often also an error about the firmware version...
[7/23/2021, 11:16:34 AM] [homebridge-philips-air] This plugin generated a warning from the characteristic 'Firmware Revision': characteristic value expected string and received undefined. See https://git.io/JtMGR for more info. [7/23/2021, 11:16:49 AM] [homebridge-philips-air] This plugin generated a warning from the characteristic 'Filter Life Level': characteristic was supplied illegal value: number 1365.3125 exceeded maximum of 100. See https://git.io/JtMGR for more info. Starting to advertise 'Homebridge 4468 456D' using bonjour-hap backend!
This is airctrl's output:
airctrl --protocol=coap --ipaddr <IP_ADDRESS>
[name]                        Name: Woonkamer
[type]                        Type: AC3033
[modelid]                     ModelId: AC3033/10
[swversion]                   Version: Ms3306
[language]                    language: EN
[DeviceVersion]               DeviceVersion: 3.0.6
[range]                       range: MarsME
[Runtime]                     Runtime: 16.43 hours
[WifiVersion]                 WifiVersion<EMAIL_ADDRESS>[ProductId]                   ProductId: 9efe665a13c111e9bc6802024953075e
[DeviceId]                    DeviceId: a87f11ca0edd11eba14ecaee0a97cbfc
[StatusType]                  StatusType: status
[ConnectType]                 ConnectType: Online
[om]                          Fan speed: turbo
[pwr]                         Power: ON
[cl]                          Child lock: False
[aqil]                        Light brightness: 100
[uil]                         Buttons light: ON
[uaset]                       uaset: A
[mode]                        Mode: T
[pm25]                        PM25: 2
[iaql]                        Allergen index: 1
[aqit]                        Air quality notification threshold: 10
[tvoc]                        Total volatile organic compounds: 1
[ddp]                         Used index: IAI
[rddp]                        rddp: 0
[fltt1]                       HEPA filter type: NanoProtect Filter Series 3 (FY2422)
[fltt2]                       Active carbon filter type: none
[fltsts0]                     Pre-filter and Wick: clean in 164 hours
[fltsts1]                     HEPA filter: replace in 3744 hours
[fltsts2]                     Active carbon filter: replace in 65535 hours
[filna]                       filna: 0
[filid]                       filid: 0
[flttotal0]                   flttotal0: 360
[flttotal1]                   flttotal1: 4800
[flttotal2]                   flttotal2: 65535

Not the same problem you are experienced but I am still having issues with the http connected AC1214/10. Running v2.3.4 and I wonder if I should update. Before there was a disclaimer that I should stay with v.2.3.4.

Not the same problem you are experienced but I am still having issues with the http connected AC1214/10. Running v2.3.4 and I wonder if I should update. Before there was a disclaimer that I should stay with v.2.3.4.

AC1214/10 not working anymore with http protocol. I don't think is good idea.

No chance to work on 3.5.2 ?? Here only problems with the new version :-((


Does NPD have a ROADMAP?

Hi, NPD community,
In k8s, npd could take over all the node monitoring, which is quite useful for cluster at large scale. We are trying to adopt npd in our production enviroment and contribute back to it. I am wondering if NPD project has a ROADMAP document to share the future work of it.
Thanks.
Allen Sun
Alibaba Cloud

There is no formal roadmap doc. The recent planned big change is to enable NPD to export metrics. You can follow up the design here: https://docs.google.com/document/d/1SeaUz6kBavI283Dq8GBpoEUDrHA2a795xtw0OvjM568/.


Revalidation triggers fetcher.submit to change

What version of Remix are you using?
1.8.2
Steps to Reproduce
Open this: https://stackblitz.com/edit/node-ikzu57?file=app%2Froutes%2Findex.tsx
Open the browser console
Click "Set searchParams"
Observe the console shows that fetcher.submit has changed.
Expected Behavior
I expect fetcher.submit to be stable so I can safely use it in dependency arrays.
Actual Behavior
I cannot use fetcher.submit in a dependency array because any time there's a revalidation on the page, all fetcher.submits are new instances of the submit function.

After further investigation, what's trigger fetcher.submit to change is the defaultAction dependency: https://github.com/remix-run/remix/blob/8aacba3c4cd23e63ef57a8cb7b68c9d6c88bdb6e/packages/remix-react/components.tsx#L1260
So, what's happening, is I make a change to the URLSearchParams, that triggers the defaultAction to change which invalidates the submit on the fetcher which triggers my useEffect again.
In my case, I don't care about the defaultAction because I specify one when I call submit.
So, in order to ensure referential stability, we need to either:

allow the user to configure the action when calling useFetcher so we can ignore the defaultAction in that case (so it won't appear in the dependency array and therefore cannot change)
Or we need to use a "latest ref" pattern so we don't have to list the defaultAction in the dependency array.

I much prefer option 1 because it would actually be a really nice feature to have anyway. It would allow folks to change their code from something like this:
const fetcher = useFetcher()

// ...
<button onClick={() => fetcher.submit({hi: 'bob'}, {method: 'post', action: '/some-path'})}>Submit bob</button>

To this:
const fetcher = useFetcher({
  method: 'post',
  action: '/some-path',
})

// ...
<button onClick={() => fetcher.submit({hi: 'bob'})}>Submit bob</button>

I've seen some folks suggest they would actually prefer this API and I think I agree with them. Also, it would solve the problem of fetcher being referentially stable.

This should be resolved by https://github.com/remix-run/react-router/pull/10336 which makes fetcher.submit stable across location changes

Should be available in Remix 1.16


Initial login and sync

Created view for initial login and sync

@freshamir We're going to need to tackle some issues going forward but this PRs just to big to tackle them all in one shot. Lets to smaller more manageable PRs going forward.


Ubershaders and pipeline pre-compilation (and dedicated transfer queues).

This is a big PR with quite a bit of history that should be evaluated very thoroughly to evaluate where do we want to make some concessions and try to mitigate the side effects as much as possible. However, the benefits are essential to shipping games with the engine and making the final experience for users much better. To read further on @reduz's notes about the topic, you can check out these documents (Part 1 and Part 2).
Due to the complexity of this PR and how 4.3 is currently in feature freeze, I'd definitely not consider this PR until 4.3 is out. If you want the TL;DR: skip ahead to the two videos with the TPS demo to see the immediate difference.
NOTE: These improvements will only affect the Forward+ and Mobile renderers. No changes are expected for Compatibility.
Transfer queues
First of all, this PR supersedes the transfer queues PR and effectively uses it as its base.  The reliance on needing to unlock parts of the behavior of RenderingDevice to make it multithread-friendly to reap the benefits was far too much to keep both PRs separate. As mentioned in that previous PR, merging it as is will cause a small performance regression unless https://github.com/godotengine/godot/pull/86333 is merged first.
Pipeline compilation
Modern APIs like Vulkan and D3D12 have made rendering pipeline management very explicit: their creation is no longer hidden behind the current rendering state and handled on demand by the driver. Instead, the developer must create the entire pipeline ahead of time and wait on a blocking operation that can take a significant amount of time depending on the complexity of the shader and the speed of the hardware. This has seen some improvements recently with the introduction of new extensions like VK_EXT_graphics_pipeline_library, but as always, Godot must engineer solutions aimed towards resolving the problem for as much hardware as possible and use such features optionally for optimization in the future.
Godot has the responsibility to perform as fast as possible for the end user, which leaves it no choice but to generate pipelines with the least amount of code and requirements as possible. The engine achieves this through the use of shader compilation macros (shader variants) and the use of specialization constants to optimize code for a particular pipeline (pipeline variants). While Godot resolves shader variant compilation and can even ship the shader cache to skip the step altogether, it coudn't resolve pipeline variant compilation ahead of time before this PR at all.
If you're familiar with the "stutters when playing the game for the first time" phenomenon that has plagued all games shipped with Godot 4's RD-based renderers, this is pretty much the entire root of the problem. This is not a problem exclusive to Godot as it's been very evident in lots of commercial releases that include very extensive shader pre-compilation steps the first time a game starts or a driver update happens. The issue is so prevalent even Digital Foundry points it out as the #1 problem plaguing PC game releases in this article and they never fail to mention the existence of the problem on any new game that suffers from it.
Ubershaders for Godot 4
The exciting part about this PR is an effective solution was developed to address this problem completely without the need to introduce extensive shader pre-compilation steps or any input from the game developer whatsoever. Instead, attempts have been made to make pipeline compilation a part of loading assets as much as possible. Not only does this mean most pipeline compilation is no longer resolved at drawing time, it can also even be done in background threads and presented as part of a regular loading screen. That means the game is no longer at the mercy of the renderer introducing these stutters when it needs to draw, but it makes the behavior much more predictable and able to be handled as part of a loading process.
The main improvement this PR makes is the introduction of ubershaders once more to the engine, but these are quite different from what was previously done in Godot 3. Unlike the previous version of the engine, these shaders do not correspond to generating text shaders with specializations and compiling them in the background, which could lead to a lot of CPU usage that'd take lots of time in weaker systems. Instead, ubershaders are mostly still very similar to the current shaders the engine already has, with a key difference: specialization constants are pulled from push constants instead. This means that the engine is able to use a version of the shader already that can be used for drawing immediately while the specialized version is generated in the background. Pipeline variants are much faster to generate like this instead of relying on runtime shader compilation to insert the constants as part of the shader text, as they work directly on the SPIR-V and skip the need to compile the shader from text again.
Specialization constants are a big part of how Godot optimizes pipelines, but they've been limited by parts of the design as to how many can actually be used. Any additional constant implied an explosion of variants that led to the pipeline cache structure getting even bigger (160 KB in just pointers in Forward+ for any single material in master at the moment!), and every new addition meant that if the state is very dynamic, stutters would occur due to extra pipeline compilation. This was quite evident in the Mobile renderer, which uses a specialization constant to disable lights if they're not used: as soon as a light popped up, then stutters due to pipeline compilation were inevitable.
With this change, a new simple hashing system for pipeline caching is introduced instead:

The required pipeline is requested in its specialized form from the cache.
If the pipeline is not available yet, compilation is started on the background without stalling the main thread.
The ubershader pipeline (which has been compiled at loading time) is used instead. The specialization constants are pushed as part of the push constant instead along with some other rasterization state parameters (like backface culling).

Pipeline compilation at loading time
The other key part behind the PR is the introduction of pipeline compilation of the ubershaders in two extra steps.

During the creation of the surface cache in the renderer (stutter on scene setup).
During loading of ArrayMeshes (can be pushed to a background thread).

The difference in how both of these changes work together is pretty evident on the TPS demo by simulating a clean run as an end user would see the first time they run the game. A big chunk of the stutters are gone, especially the one that happens the first time the character shoots, which is a typical case of a stutter that only happened at drawing time despite the effect being loaded in the scene tree already.
Both of these videos have pipeline caching disabled and the driver cache deleted between each run.
master (dc91479082bd9cd56cd74282573492fe1ba9618a)
https://github.com/godotengine/godot/assets/538504/4152255e-ee0c-409c-9fa9-06cde148a62a
transfer_and_pipelines
https://github.com/godotengine/godot/assets/538504/391eaa02-00bf-4986-8972-f1d44d80ab1b
It's also worth noting how the loading screen animation actually plays out more of the time instead of having one big stutter at the end due to the initial pipeline compilation at drawing time. These loading times are also significantly shortened by making multiple improvements to the behavior of both the shader and pipeline compiler, allowing it to multi-thread more effectively and use more of the system's resources.
The negatives (and how we can mitigate them)
As was expected, these benefits do not come for free. But there's multiple ways we can attempt to mitigate most of the extra cost and this is an area I'm open to feedback on and that we can further optimize in future versions as well.

An extra shader variant had to be introduced. @reduz has diligently always recommended against adding shader variants as it leads to a combinatory explosion, but this is one sacrifice that had to be made to allow ubershaders to exist. However, there's potential for reducing some of the existing shader variants to dynamic paths in the ubershader if possible. This is also significantly mitigated by improvements to the shader compiler's multithreading and it should be a non-issue for games that ship the SPIR-V shader cache.
Loading times are bound to be longer as pipeline compilation is pushed here instead. This is the intended effect and it's paying a cost upfront that would happen at drawing time otherwise (which is less preferable). That said, pipeline caching always plays a part here and it'll speed up loading times in later runs as it should.
Higher memory consumption from extra pipelines and shader variants being compiled that might go unused. This is sadly one cost that must be paid no matter what and can hopefully be mitigated by implementing better detection of features in use.

The biggest reason behind these negatives is the engine's flexibility. Features can be turned on and off without explicit operations from the user at a global level: a scene can be instanced to use VoxelGI while another one might use Lightmaps instead. As a matter of fact, this is exactly what the TPS demo does, so any run of the game must pre-compile the Lightmap variants because it can't know ahead of time which method the user has chosen without looking at the scene's contents, which is yet to be instanced during mesh loading.
One of the things I hope to improve while this PR is in progress is reducing the amount of variants that are pre-compiled as much as possible. Therefore it'd be great to gather feedback on which of these methods are most effective and how to implement them:

Detecting features that aren't used at the project level would help significantly. If a user never uses VoxelGI, we shouldn't be pre-compiling variants for it. The biggest culprits here that I could identify are features like separate specular and motion vectors. Adding some form of tracking somewhere at a global level so the engine can know ahead of time without having to instance scenes would be very helpful here.
Assuming features aren't enabled by default and going back to compile them if they are: this is actually something that's partially implemented with the 'advanced' shader groups already. Upgrading this to a per-feature detection could help a lot towards reducing pre-compilation and delegating it to the surface cache setup. If a developer wishes to properly delegate the pre-compilation during mesh loading, all they need to do is just instance scenes first with the appropriate features.
Allowing developers to opt in or out of variants to pre-compile at a global level instead. This is likely a very good solution for more experienced developers to fine-tune their game if it actually has a significant amount of shaders and materials that need it. This would likely be a simple set of toggles indicating which features shouldn't be compiled as the developer knows they'll never make use of them.

It's worth noting that under the current implementation, none of these leading to false positives will lead to the engine misbehaving: at worst, it just causes the drawing time stutters the current version already has.
Testing methodology

As of https://github.com/godotengine/godot/pull/90271 (which is currently merged), you can now disable the pipeline cache at the engine level (if the API supports it). It is essential to turn this setting off to do any comparisons and to simulate the experience of a clean run of a game.

Delete the driver pipeline cache. This is the last barrier of defense the driver has if the application doesn't implement a pipeline cache of its own. This heavily depends on the IHV and the platform (e.g. on Windows & NVIDIA it's located at %LocalAppData%/NVIDIA/GLCache). No test should be considered valid without deleting this cache first and foremost.

Run the game!

Trying to measure the results can be a bit tricky as the results are heavily dependent on the behavior you see in a project. As the benefits are more visually evident as seen in the videos, it is hard to measure the effects of pipeline compilation at drawing time as they present themselves as stutters that happen all throughout the game instead of one particular scenario.
New performance monitors
Some new statistics have been added to the performance monitors which should help verify without a shadow of a doubt if the pipeline pre-compilation is working as intended. There's four different pipeline compilation sources that are identified and they should help towards understanding where a extended loading time or stutter comes from.

Quoted from the documentation added by this PR:

RENDERING_INFO_PIPELINE_COMPILATIONS_MESH: Number of pipeline compilations that were triggered by loading meshes. These compilations will show up as longer loading times the first time a user runs the game and the pipeline is required.
RENDERING_INFO_PIPELINE_COMPILATIONS_SURFACE: Number of pipeline compilations that were triggered by building the surface cache before rendering the scene. These compilations will show up as a stutter when loading scenes the first time a user runs the game and the pipeline is required.
RENDERING_INFO_PIPELINE_COMPILATIONS_DRAW: Number of pipeline compilations that were triggered while drawing the scene. These compilations will show up as stutters during gameplay the first time a user runs the game and the pipeline is required.
RENDERING_INFO_PIPELINE_COMPILATIONS_SPECIALIZATION: Number of pipeline compilations that were triggered to optimize the current scene. These compilations are done in the background and should not cause any stutters whatsoever.

bugsquad edit: Fixes https://github.com/godotengine/godot/issues/61233
TODO

[x] Pass CI and address compatibility breakage (if there's any).
[x] Make sure compatibility renderer hasn't had any regressions from modifying the common classes.
[x] Address any multi-threading issues that can possibly arise from both transfer queues and this PR.
[x] Test D3D12 for any regressions or new issues introduced from multithreading.
[x] Evaluate further ways to reduce the amount of pipelines being pre-compiled.
[x] Evaluate any possible CPU-time regressions during drawing and how to mitigate them.
[x] Evaluate adding this improvement to Canvas Renderer as well.

Contributed by W4 Games. 🍀

Allowing developers to opt in or out of variants to pre-compile at a global level instead. This is likely a very good solution for more experienced developers to fine-tune their game if it actually has a significant amount of shaders and materials that need it. This would likely be a simple set of toggles indicating which features shouldn't be compiled as the developer knows they'll never make use of them.

This resembles https://github.com/godotengine/godot-proposals/issues/5229 and https://github.com/godotengine/godot-proposals/issues/6497 a lot, although I haven't proposed it for VoxelGI and LightmapGI as these are not Environment or CameraEffects properties.
If such a setting is disabled, we can assume the user is OK with having runtime shader compilation occur the first time the setting is enabled (since they'll probably be in an options menu while doing so).

Assuming features aren't enabled by default and going back to compile them if they are: this is actually something that's partially implemented with the 'advanced' shader groups already. Upgrading this to a per-feature detection could help a lot towards reducing pre-compilation and delegating it to the surface cache setup. If a developer wishes to properly delegate the pre-compilation during mesh loading, all they need to do is just instance scenes first with the appropriate features.

I gave this a shot and got pretty successful results. The current caveat is that pipeline compilation will be less likely to be triggered for resources loaded through a background thread in a loading screen unless the game features an scene first with the feature used in-place. If not, then it must defer the loading to the surface cache creation instead.
However the results are pretty good. The pre-compilation on the TPS demo has gone down significantly:

That's around 650+ pipelines reduced to around 300 pipelines, pretty much doubling the speed of the initial load in the demo that I showcased in the video and still has no pipeline stutters during drawing. I haven't detected any regressions from implementing this yet but trying to find edge cases is still worth investigating.
https://github.com/godotengine/godot/assets/538504/9a2da5c7-a8d9-4d45-af49-fb316cd43c00
I still think we could use some global settings to fine-tune the behavior (e.g. automatically detect, always pre-compile, never pre-compile), but this gets us much closer to an ideal level of pre-compilations that I wanted to see from the start.

I investigated Canvas Renderer support and the potential problems we'd have to fix to fully take advantage of it.
First off, Canvas Renderer does suffer from the exact same problem: pipelines are compiled at drawing time if necessary. However, the total amount of pipelines that this does happen on is fairly small. However, it's undeniable you can get stutters from behavior such as enabling and disabling lights in proximity of the elements.
I added the entire framework for supporting ubershaders but ultimately left it disabled for now for a few reasons even if it does work as intended.

The amount of pipelines that you can pre-compile from just the shader data is about six without taking into account the possibility of meshes and polygons. That's a lot of pipelines to pre-compile when the final amount usually ends up being far less. In one example project, the pre-compiled count was basically 24 while the specializations were merely 3. That's a lot of added loading time for very little benefit.
It's impossible for the checks I added to pre-compile pipelines ahead of time for polygons and meshes. The vertex attribute format needs to be known ahead of time with exact precision of the offsets and strides.
A real solution would involve some sort of scheme where we detect commands that get added and cached, pass those off to the renderer and pre-compile pipelines for the cached commands. However upon experimentation it seems whatever point I found that could be used as the hook was called too often to be beneficial.

For now I'm leaning towards addressing other issues the PR currently has (such as an extra CPU cost due to a mutex I want to avoid), but if anyone has an example of a project that requires lots of different shaders, pipelines, is entirely 2D and suffers from stutters, that'd help to provide a good example of something I can use as a reference.

PS: I wonder how this will interact with https://github.com/godotengine/godot/pull/88199 – does Metal make this approach possible?

PS: I wonder how this will interact with #88199 – does Metal make this approach possible?

Should be completely fine as far as I know as the entire PR is completely driver-agnostic. A lot of the changes on this one are just basically fixing a lot of stuff that wasn't thread safe, so it could expose some other bugs if part of the Metal driver assumed that wasn't gonna happen (which was a common issue in the D3D12 one but easily fixed).

Hey, great work on this! It might need a bit more testing - during my first run of the TPS demo with this branch, I got a freeze for about a second when I fired a shot (but not when it landed). This doesn't seem to be reflected in the Compilations Draw stat, so I'm not sure whether this is actually a shader compilation stutter that just wasn't accounted for, or some issue with threading or loading.
The weird thing is it seems to happen based on when I fire the shot.  If I wait a couple seconds and then fire, it works fine.  Timing seems to be related to when the flying ship enters view. This one:

Video:
https://github.com/godotengine/godot/assets/34800072/948f4c80-3915-4014-95ac-65ff1baa3c4e
Windows 11
nVidia GeForce RTX 3070
nVidia driver version 556.12

This doesn't seem to be reflected in the Compilations Draw stat

Did you check if any of the other numbers go up when you do that? Objects that don't get added to the surface cache for some reason before they're shown would still show stutters regardless of what this PR does. Surface cache compilations have their own counter shown in the monitor. Perhaps there was some behavior change that makes the bullet not get added to this surface cache before it shows up.

@KeyboardDanni Can you also show what graphical settings you used? Are you using the project unmodified? Did you change the graphical settings in the settings menu inside the game? That can also affect whether the PR encounters all cases ahead of time.

This doesn't seem to be reflected in the Compilations Draw stat

Did you check if any of the other numbers go up when you do that? Objects that don't get added to the surface cache for some reason before they're shown would still show stutters regardless of what this PR does. Surface cache compilations have their own counter shown in the monitor. Perhaps there was some behavior change that makes the bullet not get added to this surface cache before it shows up.

I can try, but the graphs only seem to be one second in resolution, which isn't enough to observe a difference when I have to fire the shot within a second or two of level start, and around that time there's more compilations going on so I don't know if any number changes might be related. But I'm open to suggestions and ideas on how to test this better. Perhaps breaking with a C++ debugger would help?

@KeyboardDanni Can you also show what graphical settings you used? Are you using the project unmodified? Did you change the graphical settings in the settings menu inside the game? That can also affect whether the PR encounters all cases ahead of time.

It's version 4.2-f0587b2 from the Asset Library. No modifications except project migration, disabling the Pipeline Cache and Shader Cache, and opening in windowed mode. Here are the graphics settings I used:

So I tested this again. Firing a shot for the first time seems to raise Compilations Surface by 2 regardless of whether the stutter occurs. Also, the length of the stutter is shorter if I wait longer to do it. So if I wait a second and a half instead of less than one second, the stutter only lasts for about half a second, almost as if it's waiting on some background processing to complete. Or maybe it has something to do with it somehow stalling until the particle system on the ship entering view? Probably a red herring though.
Video of me firing slightly later:
https://github.com/godotengine/godot/assets/34800072/568d5633-16c3-4f52-a6ed-a51f336ece34

Edit: Looks like it's caused by VoxelGI being enabled. If I use SDFGI I get no stutter after scene load. If I use LightmapGI the first frame after loading the scene takes a few seconds but otherwise I don't get any stutter. With VoxelGI, it seems like the first lighting effect forces a stutter (but only if done too early).

Thanks, when I get around to rebasing this for 4.4 I can investigate if VoxelGI is causing some pipelines to be missed.

If https://github.com/godotengine/godot/pull/94289 gets merged, a special exception will need to be made for multimesh handling.

Godot v4.3.dev (c21f06204) - Debian GNU/Linux trixie/sid trixie - X11 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3080 Ti (nvidia) - AMD Ryzen 7 5800X 8-Core Processor (16 Threads)
driver: 535.171.04
I (finally) tried running this branch because my PR #91672 seems to be blocked in favor of these changes, but it falls over on that PR's test project. Although that project tests new functionality in the Mobile renderer, it should work fine in Vulkan.
There are three major classes of issues I ran into -- note I'm on Linux/X11, but using an NVIDIA card with the most recent driver Debian unstable supports thus far.

Some shader combinations do not compile.

Under this PR, the test project from my PR fails to build shaders in Vulkan: stdout.txt.
(While the lightmap.lmbake file has a version mismatch in this PR's engine, deleting and regenerating it doesn't change the compilation situation, as expected.)

Frequent livelocks under Linux (RenderingDeviceDriverVulkan::fence_wait, stuck in driver).

After encountering the above, I tried reconstructing a simpler project from scratch, by tweaking similar attributes in the material:
emptyish.zip
, which rendered in-editor fine until I exited (see below).
After that, trying to reload the project in the editor repeatedly hits a livelock in the libnvidia-glcore.
No shader errors were reported to the console, just a skipped ICD message and Vulkan banner as seen in normal usage on my system. Here's the stack trace (in thread 1 at the bottom of the file in all these thread dumps -- on review, it seems most of the other threads are in the same state almost all the time):
bt_hang.txt.
Finally, I tried the inimitable TPS demo. I didn't see any shader compilation issues on loading, but the in-editor project load process hit similar hangs, also in the nvidia driver, waiting for a fence. Attempting to finish from the stack trace stalls indefinitely inside the driver. I had to kill the editor and reload the project three times, which finally succeeded (all resources finally imported).
bt-tps.txt
bt-tps2.txt
But then running the project led to a similar hang on the title screen once my mouse hovered over the Settings button. Same hang location, though occasionally different parent callstacks.
bt-run-tps.txt
But, there is good news. The next time I ran the project, it worked flawlessly, with similar performance wrt. expecting stuttering/loading times as shown in the videos above. I was able to play for at least a minute without issue before exiting cleanly.

Crashes on editor exit.

In a few of the times when I could run the editor and exit without killing it, I get a segfault with similar stack trace. (Maybe it is unrelated entirely, and maybe fixed since the Apr 22 baseline, but just FYI):
stdout2.txt
stdout3.txt

While testing these situations, I deleted the .godot folders between each run, by necessity due to differences in the baselines, and later also tried deleting the Nvidia GL caches (seems to have evolved from ~/.nv/GLCache -> ~/.cache/nv/GLCache -> ~/.cache/nvidia/GLCache on my system0 and turned off the Pipeline Cache setting just in case there is some relation, but it doesn't affect these results. I didn't  delete .godot or clear shaders after finding the repeated source of the livelock (except when swapping engines) since that only exacerbated the hangs.
Needless to say, all the projects load fine without shader issues or hangs in 4.3 stable. TPS Demo from clean project, run, and editor exit is summed up in:
bt-tps-stable.txt
I hope these turn out to be relatively simple issues to resolve in this branch. This is very exciting work!

Pushed a big rebase, will re-run tests and verifications soon before marking it as ready for review.

Edit: Looks like it's caused by VoxelGI being enabled. If I use SDFGI I get no stutter after scene load. If I use LightmapGI the first frame after loading the scene takes a few seconds but otherwise I don't get any stutter. With VoxelGI, it seems like the first lighting effect forces a stutter (but only if done too early).

I took a look at this and I'm unable to find any reason why SDFGI and VoxelGI would behave different. In fact, SDFGI has even more pipeline compilations when the shooting is triggered on surface: VoxelGI goes up by 2 and SDFGI goes up by 4.
However, Clay and I found the following information about it.

Player only instances the bullet when you shoot, which means bullet.tscn is never loaded into the scene ahead of time. Pipelines are compiled when the object is instanced. https://github.com/godotengine/tps-demo/blob/34b6977f3524a0ad513bda0bc06bea27e795c154/player/player.gd#L132
Preloading the scene doesn't seem to help, most likely because loading the mesh resource itself isn't enough information to find the pipeline ahead of time. This can happen for a multitude of reasons which we're trying to determine the possible reasons as to why yet.
The delta of pipelines is very small, which hardly sounds like a new surface getting added is triggering a lot of these pipelines.

I'll keep investigating the possible reason, but the problem does go away if an invisible bullet scene is instanced into the tree ahead of time. That said, I can't reproduce the stutter myself, but I do see the pipelines increasing at least.

The delta of pipelines is very small, which hardly sounds like a new surface getting added is triggering a lot of these pipelines.

Indeed the problem is just the fact that an effect is not enabled by the time it preloads the bullet scene, so it doesn't create the pipelines necessary for either SDFGI or VoxelGI.
This is the problem I brought up that could happen before.

The current caveat is that pipeline compilation will be less likely to be triggered for resources loaded through a background thread in a loading screen unless the game features an scene first with the feature used in-place. If not, then it must defer the loading to the surface cache creation instead.

I get what's going on now.
TPS Demo has this on the title screen.

This is an attempt by the game to cache the pipelines ahead of time for the bullet. However, the title screen does not use SDFGI or VoxelGI, so the ubershader pipeline pre-compilation is not detecting it should create these pipelines.
The extra two to four pipelines we're seeing on the first shot are the new pipelines encountered due to the fact the effect only gets enabled when starting the game, but the instancing of this bullet ahead of time is not done there instead.
Merely instancing the bullet scene as hidden would fix this issue immediately, as hidden surfaces are added to the cache.

I've tested locally with the TPS demo and the Legend of the Nuku Warriors demo
TPS Demo
I can reproduce the issue that Calinou flagged above with the core particles turning opaque. I can reproduce the issue in the editor by going to the level scene and flying the camera into the core. If I go to the core scene the issue goes away. Same if I make the children editable in the level scene, then hide the particles and show them again.
I cannot reproduce the crash
All cases of stuttering are gone for me in this demo (with the bullet loading fix from Dario)
Nuku
When running the Nuku warriors demo I get a deadlock during scene loading. I tested 2be730a05b7ff221b89c967981f7caee6e164ef0 (the commit before this one) and I get the same deadlock. So that is an existing problem in master

I got a crash once when opening the TPS demo in the editor. I could only reproduce this once after removing ~/.cache/nvidia/. I couldn't reproduce this again after trying to follow the same steps.

@Calinou Your crash in particular seems to be on the shader compiler, so taking out your SPIR-V cache seems like the most likely way to reproduce it. That said, it seems to be quite a few levels deep down. I've not reviewed the possibility of glslang having some shared state between threads yet that could lead to corruption but it shouldn't be the case.

The transparency issue should be fixed. There was a local variable to the material that was only being updated during pipeline creation, introducing a possible race condition if the draw list happened to insert it first. This has been moved to only be updated on set_code as it was before.

I still get a big stutter when I first turn around, but all the other stutters are gone.

So far this one seems to be just the fact that the usage of light cubemaps is not triggered until the camera is turned around. A bit of a content issue but perhaps we can mitigate these by being able to know ahead of time from some other means whether light cubemaps or atlases are used in some other way. Perhaps light storage could keep track of this information somehow or notify the renderer when a new shadow mode is used? Or just keep track of the generated lights altogether and we use the dependencies to know?

I've improved the light detection for variants to be ahead of time by making light storage toggle these variants when the shadow types are set. This gets rid of stutters when the light variant is not in view yet on the scene.

================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.4.dev.custom_build (fdac73548424c59899cb8cde26c551570107ebca)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] /lib64/libc.so.6(+0x40d00) [0x7facce5c5d00] (??:0)
[2] bin/godot.linuxbsd.editor.x86_64() [0xa82fb2] (/usr/include/c++/14/bits/char_traits.h:427)
[3] bin/godot.linuxbsd.editor.x86_64() [0xabe70a] (/home/hugo/Documents/Git/godotengine/godot/thirdparty/glslang/glslang/MachineIndependent/Scan.cpp:933)
[4] bin/godot.linuxbsd.editor.x86_64() [0xac1850] (/home/hugo/Documents/Git/godotengine/godot/thirdparty/glslang/glslang/MachineIndependent/Scan.cpp:296 (discriminator 2))
[5] bin/godot.linuxbsd.editor.x86_64() [0xb0049b] (/home/hugo/Documents/Git/godotengine/godot/thirdparty/glslang/glslang/MachineIndependent/ParseHelper.cpp:199)
[6] bin/godot.linuxbsd.editor.x86_64() [0x99098a] (/home/hugo/Documents/Git/godotengine/godot/thirdparty/glslang/glslang/MachineIndependent/ShaderLang.cpp:1212)
[7] bin/godot.linuxbsd.editor.x86_64() [0x3bde5a4] (/home/hugo/Documents/Git/godotengine/godot/./servers/rendering/rendering_device.cpp:215)
[8] bin/godot.linuxbsd.editor.x86_64() [0x3db4338] (/home/hugo/Documents/Git/godotengine/godot/./core/templates/cowdata.h:461)
[9] bin/godot.linuxbsd.editor.x86_64() [0x48fbea6] (/home/hugo/Documents/Git/godotengine/godot/./core/object/worker_thread_pool.cpp:101)
[10] bin/godot.linuxbsd.editor.x86_64() [0x48fc6d0] (/home/hugo/Documents/Git/godotengine/godot/./core/object/worker_thread_pool.cpp:205)
[11] bin/godot.linuxbsd.editor.x86_64() [0x42c9e3d] (/home/hugo/Documents/Git/godotengine/godot/./core/os/thread.cpp:64)
[12] bin/godot.linuxbsd.editor.x86_64() [0x51558a4] (thread.o:?)
[13] /lib64/libc.so.6(+0x976d7) [0x7facce61c6d7] (??:0)
[14] /lib64/libc.so.6(+0x11b60c) [0x7facce6a060c] (??:0)
-- END OF BACKTRACE --
================================================================
[1]    53286 IOT instruction (core dumped)  bin/godot.linuxbsd.editor.x86_64 --path ~/Documents/Godot/tps-demo -e

I'm not really able to gather much about this one. The crash would imply glslang has some internal error but a brief review about it gives me no shared context whatsoever that could be causing the issue. It'd be no different of a situation from what was there before this PR, it's just that this PR is able to parallelize shader compilation more effectively.

I've made https://github.com/godotengine/godot/pull/97465 as a new related PR to avoid a performance regression from this PR.

Not sure if you missed my comment, but I'm still seeing the mostly reliable livelock behavior as I mentioned before, with all the same kinds of stack traces. The fade test project I linked uses a lot of different materials in one scene and seems to trigger (threading?) issues quite easily.
Updated stacks:
stack.txt (in thread 1)
stack2.txt (in thread 10)
Occasionally I can run properly, and when editing or running that project, this PR spits out a repeatable series of errors (while seeing to render properly nonetheless):
Godot Engine v4.4.dev.custom_build.ccb0fb316 (2024-09-25 16:53:21 UTC) - https://godotengine.org
WARNING: GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
	terminator_CreateInstance: Received return code -3 from call to vkCreateInstance in ICD /usr/lib/x86_64-linux-gnu/libvulkan_virtio.so. Skipping this driver.
	Objects - 1
		Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle<PHONE_NUMBER>6768
     at: _debug_messenger_callback (drivers/vulkan/rendering_context_driver_vulkan.cpp:639)
Vulkan 1.3.242 - Forward+ - Using Device #0: NVIDIA - NVIDIA GeForce RTX 3080 Ti

ERROR: Mismatch fragment shader output mask (0) and framebuffer color output mask (1) when binding both in render pipeline.
   at: render_pipeline_create (servers/rendering/rendering_device.cpp:3452)
ERROR: Mismatch fragment shader output mask (0) and framebuffer color output mask (1) when binding both in render pipeline.
   at: render_pipeline_create (servers/rendering/rendering_device.cpp:3452)
ERROR: Condition "pipeline.is_null()" is true.
   at: _create_pipeline (servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:381)
ERROR: Condition "pipeline.is_null()" is true.
   at: _create_pipeline (servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:381)
ERROR: This render pipeline requires (0) bytes of push constant data, supplied: (32)
   at: draw_list_set_push_constant (servers/rendering/rendering_device.cpp:4298)
ERROR: No render pipeline was set before attempting to draw.
   at: draw_list_draw (servers/rendering/rendering_device.cpp:4319)
ERROR: Mismatch fragment shader output mask (0) and framebuffer color output mask (1) when binding both in render pipeline.
   at: render_pipeline_create (servers/rendering/rendering_device.cpp:3452)
ERROR: Mismatch fragment shader output mask (0) and framebuffer color output mask (1) when binding both in render pipeline.
   at: render_pipeline_create (servers/rendering/rendering_device.cpp:3452)
ERROR: Condition "pipeline.is_null()" is true.
   at: _create_pipeline (servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:381)
ERROR: Condition "pipeline.is_null()" is true.
   at: _create_pipeline (servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:381)
ERROR: This render pipeline requires (0) bytes of push constant data, supplied: (32)
   at: draw_list_set_push_constant (servers/rendering/rendering_device.cpp:4298)
ERROR: No render pipeline was set before attempting to draw.
   at: draw_list_draw (servers/rendering/rendering_device.cpp:4319)
ERROR: The vertex format used to create the pipeline does not match the vertex format bound.
   at: draw_list_draw (servers/rendering/rendering_device.cpp:4326)
ERROR: The vertex format used to create the pipeline does not match the vertex format bound.
   at: draw_list_draw (servers/rendering/rendering_device.cpp:4326)
ERROR: This render pipeline requires (16) bytes of push constant data, supplied: (32)
   at: draw_list_set_push_constant (servers/rendering/rendering_device.cpp:4298)
ERROR: The vertex format used to create the pipeline does not match the vertex format bound.
   at: draw_list_draw (servers/rendering/rendering_device.cpp:4326)
ERROR: This render pipeline requires (16) bytes of push constant data, supplied: (32)
   at: draw_list_set_push_constant (servers/rendering/rendering_device.cpp:4298)
ERROR: The vertex format used to create the pipeline does not match the vertex format bound.
   at: draw_list_draw (servers/rendering/rendering_device.cpp:4326)
^C

Debian GNU/Linux trixie/sid trixie - X11 - Vulkan (Mobile) - dedicated NVIDIA GeForce RTX 3080 Ti (nvidia) - AMD Ryzen 7 5800X 8-Core Processor (16 Threads)

Not sure if you missed my comment, but I'm still seeing the mostly reliable livelock behavior as I mentioned before, with all the same kinds of stack traces. The fade test project I linked uses a lot of different materials in one scene and seems to trigger (threading?) issues quite easily.

I haven't missed it, but a lot of the information in the comment did not seem very related to the PR and it was done on a version several months behind of master. I feel a lot of those need a lot of further validation of whether something is related to it or not
I do however suspect the one where pipelines are unable to be created is worth a look. I'll take a look at the test project specifically for that.
But I've done pretty extensive testing on the threading logic and most of it is managed by the WTP class of the engine, so I don't really get much use out of extensive stack traces and have little control over it from this PR's scope if there's issues with its management. It might be an area you'll need to investigate yourself if I can't find much else about it.

It'd be great if there were some sort of Resource or ProjectSetting that could be used to fine-tune pre-compilation. For example, if you know that you'll need VoxelGI or certain light types in your scene, you could specify that directly.

There's most likely plans to add something like this to aid packing of pre-compiled shaders into exported projects, so such a setting will most likely be borrowed from. It'll most likely be done after this PR though.

I haven't missed it, but some the information in the comment did not seem very related to the PR and it was done on a version several months behind of master. I feel some of those reports need further validation of whether something is related to the PR or not.

I did run against master and the project works fine in all ways. I wouldn't have reported the issue against this PR otherwise.

I did run against master and the project works fine in all ways. You've rebased against master as well, so only these changes are likely involved. I wouldn't have reported the issue against this PR otherwise. :)

Yes, but quite a few of the errors were caused from something else in the project already failing to compile and the PR being unable to handle the failure case silently. While we shouldn't have regressions in behavior like that and the engine should keep working under those conditions, I did handle most of them already by the virtue of just not crashing with an assert.
But I can't fix the shader compilation errors on the project itself and those are most likely not related to the PR. It'd be best for content to not rely on errors behaving the same between major changes, as it's not exactly guaranteed we'll handle all those errors silently in the future. By compilation errors I mean stuff like:
ERROR: Failed parse:
ERROR: 0:63: '*' :  wrong operand types: no operation '*' exists that takes a left-hand operand of type 'layout( location=0) smooth out highp 3-component vector of float' and a right operand of type ' temp highp 2-component vector of float' (or there is no acceptable conversion)
ERROR: 0:63: '+' :  wrong operand types: no operation '+' exists that takes a left-hand operand of type ' temp highp 2-component vector of float' and a right operand of type 'layout( location=0) smooth out highp 3-component vector of float' (or there is no acceptable conversion)
ERROR: 0:63: 'assign' :  cannot convert from ' temp highp 2-component vector of float' to 'layout( location=0) smooth out highp 3-component vector of float'
ERROR: 0:63: '' : compilation terminated
ERROR: 4 compilation errors.  No code generated.
   at: _compile_variant (servers/rendering/renderer_rd/shader_rd.cpp:299)
ERROR: code:
   1 |
   2 | #version 450
   3 |
   4 |
   5 |
   6 | #define USE_MULTIVIEW
   7 |
   8 | #define RENDER_DRIVER_VULKAN
   9 |
  10 | #ifdef USE_MULTIVIEW
  11 | #ifdef has_VK_KHR_multiview
  12 | #extension GL_EXT_multiview : enable
  13 | #define ViewIndex gl_ViewIndex
  14 | #else
  15 | #define ViewIndex 0
  16 | #endif
  17 | #endif
  18 |
  19 | #define FLAG_FLIP_Y (1 << 0)
  20 | #define FLAG_USE_SECTION (1 << 1)
  21 | #define FLAG_FORCE_LUMINANCE (1 << 2)
  22 | #define FLAG_ALPHA_TO_ZERO (1 << 3)
  23 | #define FLAG_SRGB (1 << 4)
  24 | #define FLAG_ALPHA_TO_ONE (1 << 5)
  25 | #define FLAG_LINEAR (1 << 6)
  26 | #define FLAG_NORMAL (1 << 7)
  27 | #define FLAG_USE_SRC_SECTION (1 << 8)
  28 |
  29 | #ifdef USE_MULTIVIEW
  30 | layout(location = 0) out vec3 uv_interp;
  31 | #else
  32 | layout(location = 0) out vec2 uv_interp;
  33 | #endif
  34 |
  35 | layout(push_constant, std430) uniform Params {
  36 | 	vec4 section;
  37 | 	vec2 pixel_size;
  38 | 	float luminance_multiplier;
  39 | 	uint flags;
  40 |
  41 | 	vec4 color;
  42 | }
  43 | params;
  44 |
  45 | void main() {
  46 | 	vec2 base_arr[4] = vec2[](vec2(0.0, 0.0), vec2(0.0, 1.0), vec2(1.0, 1.0), vec2(1.0, 0.0));
  47 | 	uv_interp.xy = base_arr[gl_VertexIndex];
  48 | #ifdef USE_MULTIVIEW
  49 | 	uv_interp.z = ViewIndex;
  50 | #endif
  51 | 	vec2 vpos = uv_interp.xy;
  52 | 	if (bool(params.flags & FLAG_USE_SECTION)) {
  53 | 		vpos = params.section.xy + vpos * params.section.zw;
  54 | 	}
  55 |
  56 | 	gl_Position = vec4(vpos * 2.0 - 1.0, 0.0, 1.0);
  57 |
  58 | 	if (bool(params.flags & FLAG_FLIP_Y)) {
  59 | 		uv_interp.y = 1.0 - uv_interp.y;
  60 | 	}
  61 |
  62 | 	if (bool(params.flags & FLAG_USE_SRC_SECTION)) {
  63 | 		uv_interp = params.section.xy + uv_interp * params.section.zw;
  64 | 	}
  65 | }
  66 |
  67 |
  68 |
   at: _compile_variant (servers/rendering/renderer_rd/shader_rd.cpp:302)

Which seems highly unlikely that this doesn't happen in master as it affects parts of the GLSL the PR did not modify at all.
But like I said, I'm interested in the latest behavior that is repeatable on the latest version, so I'll check out the project and compare against master to be sure, I just did not prioritize it due to the nature of the errors cropping up from the content itself relying on the renderer handling these errors silently.

I pushed a few fixes that affected scenes that only used lightmaps and not other effects, which was the edge case the test project might've falled into. I looked into fixing it first through another project that fell in a similar scenario so it was most likely that.

I pushed a few fixes that affected scenes that only used lightmaps and not other effects, which was the edge case the test project mentioned before might've fell into. I looked into fixing it first through another project that fell in a similar scenario so it was most likely that.

Yup, all the reported errors are gone, thanks!

For folks keeping up to date with this PR, we encountered a few problems that currently make this a bit risky to merge with the Metal backend. I'm unsure at the moment if the problem originates from the PR itself or just the fact that the Metal backend was not made to go through such heavily multithreaded work on the past before.
As far as I'm concerned I consider the PR to be done and it's been stable for us in Windows and Linux so far, but keep it in mind if Mac support is important for you.

Small update, it seems most of it was related to the stack size default being a different size and a bit too small for Godot. Increasing this has fixed most of the crashes. I'm still tracking one remaining issue but the PR seems to be working fine now on Mac.

Anyone wanting a brief introduction to this merge can have a look at the tutorial introduced by the PR to the docs here: https://github.com/godotengine/godot-docs/pull/10007

Super excited to try this with the XR Editor in Dev 4. Bit of a question however - Does this support the compatibility renderer?

Does this support the compatibility renderer?

I'm afraid it's pretty much not by design. Modern APIs like Vulkan are the only ones that provide direct control over creating pipelines, which is what this entire system is designed around.

This PR could've fixed https://github.com/godotengine/godot/issues/95112 , needs further testing.


discuss IMPORT with pg_dump

https://www.cockroachlabs.com/docs/stable/import-data.html#import-from-postgresql-dump could potentially be improved to show how to use IMPORT with pg_dump. This involves moving the create table statement to a separate file, or rather probably invoking pg_dump in a data-only mode or something. Also comes with a caveat that it only works in cockroach on new tables.

@mjibson, does this apply to 1.2 only, or also 1.1?

@Amruta-Ranade, since you've taken over docs for IMPORT, assigning this to you.

This can be done for 1.1


[SC-876] Support UniswapV3Like oracles

Check PR before merge and after check

Codecov Report
Patch coverage: 100.00% and project coverage change: +0.12 :tada:

Comparison is base (f00bc55) 77.36% compared to head (97fd0ab) 77.49%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #72      +/-   ##
==========================================
+ Coverage   77.36%   77.49%   +0.12%
==========================================
  Files          26       26
  Lines         698      702       +4
  Branches      152      152
==========================================
+ Hits          540      544       +4
  Misses        158      158

Impacted Files
Coverage Δ

contracts/oracles/UniswapV3LikeOracle.sol
98.14% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

fixes #70


add window which shows running modes

fix #6

I like it.  I see you went the simple route and pass the entire active mode list with every mode event.  That way there is no need to maintain state.  What about adding priority to the list as it may be useful for debugging as well since so many things are tied to mode priority?


InMemorySink doesn't see log entries

I have a plugin architecture where a half dozen plugins correctly find InMemorySink.Instance containing log entries...as expected.  However, I have one plugin that uses the exact same code...but InMemorySink.Instance is empty.  The screenshot below shows that the inmemory sink has indeed collected entries, but the static access point doesn't see them.
At the point where I call CreateLogger, I have run the following identity checks:
var assemblies = AppDomain.CurrentDomain.GetAssemblies().Where(x => x.GetName().Name.Contains("Serilog")).Select(x => x.Location);
var inmem = AppDomain.CurrentDomain.GetAssemblies().First(x => x.FullName.Contains("InMemory")).GetHashCode();
var hash = InMemorySink.Instance.GetHashCode();

When I run those same three checks at the location seen in the screenshot below, the results are as follows:

The assembles var shows the InMemory assembly location is the same in both places.
The inmem var shows that the hash code of the InMemory assembly is the same in both places.
The hash var shows that the static InMemorySink.Instance has indeed changed values!

I conclude from this that there is no funny business regarding mismatched DLL's or DLL versions.  Instead it seems that internally, InMemory is somehow overwriting its static value.  This seems like a bug.
Please help. :)

Hi @brentarias!
Internally the InMemorySink uses an AsyncLocal<T> to store its instance. Is the one plugin that doesn’t find the right instance perhaps initialised in an async context?

I have an update.  I found a workaround.
I'm using InMemorySink.Instance within the context of my MSTest automation.  All of the MSTest automation routines are declared async.  I became suspicious that InMemorySink.Instance might have tied object lifetime to a particular thread.  So within my [TestInitialize] MSTest setup, immediately after configuring my dependency injection and logging, I grab and store InMemorySink.Instance into a class property.  When all the actual tests run, each then uses that variable...and the problem was solved!
To say this differently, I chose to syphon off the InMemorySink.Instance value before any async/await routines were entered.

To answer your question, the initialization is enacted within a unit test framework:
        [TestInitialize]
        public async Task Initialize() { // bla, bla, bla, blah }

So yes, I could say it is in an async context.  All the test methods are also async.  It is within the above method that I have chosen to "pre-capture" the InMemorySink value, before it becomes "corrupt" or lost.

Ok that's indeed for MSTest the approach I usually take as well.
Glad you already found it.

As of now I discovered that if I simply change the signature to remove async from my [TestInitialize] method...the problem goes away!  None of my other plugins had async on their [TestInitialize] routine.  So now I know why only one plugin was affected.

Still good to know, I’ll see if I can do something about that because I imagine more people might bump into this

All of my plugins have an associated plugin test project.  All of those test projects use async test methods.  So if you want to reproduce the problem, be sure to have an async test method joined with an async [TestInitialize] method.

My solution was to wait before checking the log.
 await Task.Delay(100); // Giving time for logs to flush.
 var informationMessages = InMemorySink.Instance.Should()
        .HaveMessage("HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed:0.0000} ms")
        .WithLevel(LogEventLevel.Information)
        .Appearing().Times(2);

The real problem here is that InMemorySink is not thread-safe. Providing a static singleton instance is not really a good decision because it's possible for unit tests (or production code, for that matter) to be run in parallel depending on the library and configuration. This will lead to non-deterministic state inside Instance.
Without improvements to InMemorySink's API, a better way to use it is like this:
InMemorySink inMemorySink = new();
LoggerConfiguration loggerConfiguration = new();

loggerConfiguration.WriteTo.Sink(inMemorySink);

ILogger logger = loggerConfiguration.CreateLogger();

logger.Information("Hello, World!");

inMemorySink.LogEvents.Should().ContainSingle(a => a.MessageTemplate.Text == "Hello, World!");


Chrome binary not found error

Hi
I am getting the following error:
I have downloaded and given 755 and +x permission to chrome binary. I am not sure what is the problem. Chrome drivers is in the same folder as well.
Traceback (most recent call last):
File "lazyshot.py", line 80, in
result = takeScreeshot(url)
File "lazyshot.py", line 70, in takeScreeshot
driver = webdriver.Chrome(executable_path=os.path.abspath("chromedriver"), chrome_options=chrome_options)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 75, in init
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in init
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.9.248316,platform=Linux 4.12.0-kali1-686-pae x86)

I'm happy to get rid of this problem.
Can you tell me the output of command chromedriver --v?

Hi
root@justpentest:~/Desktop/tools/lazyshot# ./chromedriver --v
Starting ChromeDriver (v2.9.248316) on port 9515
root@justpentest:~/Desktop/tools/lazyshot# uname -a
Linux justpentest 4.12.0-kali1-686-pae #1 SMP Debian 4.12.6-1kali6 (2017-08-30) i686 GNU/Linux

It seems like you are not added the PATH in .bash_profile, file please check the documentation. chromedriver command should be work globally.

I have set bash_profile and I am able to execute chromedriver from any folder
root@justpentest:/etc# chrome  [tab][tab]
chrome-gnome-shell        chromedriver              chromedriver_linux32.zip  chromeos-tpm-recovery
root@justpentest:/etc# chromedriver
Starting ChromeDriver (v2.9.248316) on port 9515
^C
root@justpentest:/etc#
THe command"chromedriver" is working globally.

I'm not able to reproduce this issue. I think it's a client-side issue. You can check here if you missing something while installing.


Modified keys in ruby armour recipes to map correctly

Hi,
Thanks for this mod, I'm really enjoying it.
I was having an issue with the ruby armour where it was uncraftable in both the 20w16a and 20w17a versions, the logs indicated that the issue was in the recipe files, it looks like the keys just didn't match the crafting recipe.
Thanks

Hey, sorry about, mistake from moving too fast.  updated will be pushed shortly.
What do you think about a set of obsidian tools and armor being added to this mod?

No worries, I've done the same thing way too many times.
That's a great idea! What are you thinking in terms of strength in relation to the other materials?

maybe place obsidian gear above steel, but less than emerald

I made the recipe mistake in 1.15.2 as well, updating.  Thanks for letting me know.

Both version now updated.

Yeah I think that would make the most sense, obviously the blocks are hard but getting them is definitely easier than emerald, and okay great, thanks again for the mod and quick response!

manually changed


[2.4.3] Error: apiVersion 'v2' is not valid. The value must be "v1" : exit status 1 on Helm 3 Catalog

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):
The version of helm to create the charts in the next steps is:
./helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}

1 - Create a new application using helm: ./helm create rancher-v3-issue. It should directly create a Helm 3, but ensure in the Charts.yaml that apiVersion is set to v2.
2 - Create a new git repository and place that application in it. By default the version is 0.1.0 so the current layout for me was:
.
└── charts
    └── rancher-v3-issue
        └── 0.1.0
            ├── charts
            ├── Chart.yaml
            ├── templates
            │   ├── deployment.yaml
            │   ├── _helpers.tpl
            │   ├── hpa.yaml
            │   ├── ingress.yaml
            │   ├── NOTES.txt
            │   ├── serviceaccount.yaml
            │   ├── service.yaml
            │   └── tests
            │       └── test-connection.yaml
            └── values.yaml

3 - Add a new catalog in rancher using that git repository. Make sure to select Helmv3. Then deploy the application. No issue should be encountered
4 - Duplicate the 0.1.0 directory into 0.1.1 and edit the Chart.yaml file to reflect the change of value. In my case:
.
└── charts
    └── rancher-v3-issue
        ├── 0.1.0
        │   ├── charts
        │   ├── Chart.yaml
        │   ├── templates
        │   │   ├── deployment.yaml
        │   │   ├── _helpers.tpl
        │   │   ├── hpa.yaml
        │   │   ├── ingress.yaml
        │   │   ├── NOTES.txt
        │   │   ├── serviceaccount.yaml
        │   │   ├── service.yaml
        │   │   └── tests
        │   │       └── test-connection.yaml
        │   └── values.yaml
        └── 0.1.1
            ├── charts
            ├── Chart.yaml
            ├── templates
            │   ├── deployment.yaml
            │   ├── _helpers.tpl
            │   ├── hpa.yaml
            │   ├── ingress.yaml
            │   ├── NOTES.txt
            │   ├── serviceaccount.yaml
            │   ├── service.yaml
            │   └── tests
            │       └── test-connection.yaml
            └── values.yaml

And the diff:
Common subdirectories: 0.1.0/charts and 0.1.1/charts
diff 0.1.0/Chart.yaml 0.1.1/Chart.yaml
18c18
< version: 0.1.0
---
> version: 0.1.1
23c23
< appVersion: 1.16.0
---
> appVersion: 1.16.1
Common subdirectories: 0.1.0/templates and 0.1.1/templates

5 - Push the change to git and refresh the catalog
6 - Update the already deployed application to 0.1.1. This should work smoothly as the helmversion is kept during upgrade.
7 - Deploy the application again (fresh one, no upgrade) but select directly 0.1.1 as version. This will fail with the following message:
Wait helm template failed. Error: apiVersion 'v2' is not valid. The value must be "v1" : exit status 1 . Looking at the API view for the application, the helmVersion isn't set at all in that case, and it seems that rancher fallback to Helm 2 in that case even if the catalog is set to Helm v3.
8 - Delete the catalog and recreate it. The deployment of the 0.1.1 will work directly using Helm v3
Other details that may be helpful:
It can be workaround by using a version that already exist in the catalog at the time of the initial import then upgrade. But this isn't really viable in the long term. Seems similar to https://github.com/rancher/rancher/issues/26407, but we are not in a migration use case here as the catalogue was created after upgrading to 2.4.3
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):2.4.3
Installation option (single install/HA): HA

gz#10141

Same here, starting with empty chartmuseum I managed somehow to create helm chart with apiversion v2 which can be deployed. But just increasing version in Chart.yaml and uploading the same chart, results in error message as above: 2020/06/11 13:22:56 [ERROR] AppController p-kpg44/tiddlywiki-dev [helm-controller] failed with : wait helm template failed. Error: apiVersion 'v2' is not valid. The value must be "v1"

Same issue in rancher 2.4.4
Seems the only the initial version would work correctly

Same issue in rancher 2.4.4. Deleting an re-importing the helm 3 repo seems to fix the issue for now.
This happened when i tried to deploy an app from one Project to another.

This is happening for us every time using a private repo (Nexus v3) with our charts with v2 api - removing and adding again doesn't seem to make any difference. Rancher 2.4.5, the catalog is global, installing into a project on an EKS cluster

The bug is reproduced in Rancher:v2.4.5, the steps are the same as the issue provides.
the existing app can be upgraded, but deploying a new app with the new chart version fails

the bug fix is validated in Rancher:v2.4-3304-head and Rancher:master-3301-head. Following the same steps, now the app can be deployed successfully.

View the app in API, we can see the helmVersion is set properly
helmVersion": "helm_v3",

@jiaqiluo will upgrading fix existing versions of the chart metadata that was synced by versions affected by this bug?

@dasMulli  I don't think updating will fix the existing app which is in a bad state caused by this bug. You need to redeploy the app.

thanks, so deploying a new app is enough or worst case removing and re-adding the catalog?

The fix is at the catalog level, so re-adding the catalog will be the ultimate workaround if other means do not work.


Question about storing models in Container

Hi,
Sorry if this is obvious : ) but, I'm trying to build the Docker container. It says to "First, set the MODEL_PATH and SESSIONS_PATH variables in the .env file to the actual directories on the host." What I want to do is build the container with one or a few models stored in the container, then run the container on a cloud gpu. So in that case, what should I put as the MODEL_PATH and SESSION_PATH, can I just create a /model directory in the container and story the models in there and then just point the MODEL_PATH var ro /models/(my downloaded model)
APPLICATION_STATE_PATH=/data  # path to the directory holding application state inside the container
MODEL_PATH=F"models/{MY_MODEL}"  # replace with the actual model path on the host
SESSIONS_PATH=~/exllama_sessions  # replace with the actual directory on the host where chat sessions should be stored

I'm sorry I really don't know anything about docker. @nopperl did the Docker stuff, maybe they can help?

@JacobGoldenArt in the provided docker compose setup, the model is not stored in the container! Instead a host directory is mounted into the container. Also, exllama expects the directory to contain a single model instead of multiple models.
So, in your case, you could have a /models directory on the host which contains all your models. You would then start the container with a specific model (e.g. MODEL_PATH=/models/LLaMA-7B-4bit-128g). If you want to switch to a different model, restart the container with a different MODEL_PATH.


Flask doesn't auto-escape files not ending in html, htm, xml, xhtml

    def select_jinja_autoescape(self, filename):
        """Returns ``True`` if autoescaping should be active for the given
        template name. If no template name is given, returns `True`.

        .. versionadded:: 0.5
        """
        if filename is None:
            return True
        return filename.endswith(('.html', '.htm', '.xml', '.xhtml'))

This seems odd. Why not just autoescape all file extensions. Or at least include .j2 and .jinja2?

Because not all file extensions need HTML escaping, and there are no standard file extensions for Jinja. From Jinja's docs: "A Jinja template doesn't need to have a specific extension: .html , .xml , or any other extension is just fine."
You can override this method if you would like to autoescape a different set of files.

There may be no accepted standard file extension to the pallets but users are definitely using extensions that aren't going to be escaped but rendered to the browser.
Why not just include values that users are using? In what context of flask are people using render_template or render_template_string not expecting that the content will be escaped?
This is quite a serious gotcha especially if someone decides to use an extension that is used elsewhere (Ansible for j2).

Ansible might use .j2, but I've rarely seen it in the flask world. All flask examples and tutorials i know use .html

I mean this was the second result on Google: https://github.com/alex-leonhardt/flask-blog/blob/master/templates/post.html.j2
And if a user is curious and decides to search what others are using as file extensions it shows that others are using .j2. And it isn't clear that this causes them to be trivially vulnerable to XSS.
Perhaps state in the documentation that .html is the correct template extension?

was infuriated by this setting when a security researcher notified me about XSS in my site.
should at least warn users on the first page of the documentation.

the solution is to subclass Flask and replace the method with your own extensions.


Metadata from Volume is not included in Volume Snapshot

Hi guys,
as part of an internal proof of concept, we're evaluating Velero to backup cluster resources and volumes. The volume snapshots are successfully created in OpenStack and visible. Restoring them also works properly. We really appreciate the work you put into this plugin.
We have noticed, that the metadata from volumes is not included in the backup.
See this example of Metadata:

In the restored volume, Metadata is 'none'. Is there any way to extend capabilities to include this? It's more of a nice-to-have, as it makes mapping volumes to cluster resources easier.
Best regards,
Jacob

Hi @lappesj,
Thanks for filling an issue. I think this is valid idea and also technically doable.
I will look into that.

The metadata should be fixed in the #81


Suggestion: Generic Parameter Overloads

Suggestion moved over from codeplex.

The suggestion is to allow generic parameter overloads like the following:
declare module Backbone {
    class Model{}
    class Events{}
    class ViewOptions<TModel extends Model> {}
    class Collection<TModel extends Model> {}

    // Generic parameter overload
    class View extends View<Backbone.Model>;
    class View<TModel extends Model> extends Events { // Don't take as duplicate identifier
        constructor(options?: ViewOptions<TModel>);
        model: TModel;
        collection: Collection<TModel>;
    }
}

// With Generics Parameter overloads, old non-generic consumer
// code works as before and does not need to change.
class MyView extends Backbone.View {
}
var myView = new MyView();
var model = myView.model; // model is of type Backbone.Model

// Can gradually introduce generic parameters to Views
class DerivedModel extends Backbone.Model {
}
var myView2 = new Backbone.View<DerivedModel>();
var model2 = myView2.model; // model2 is of type DerivedModel

This is important because this would allow people to introduce generics in their library declarations without breaking large amounts of existing code for the consumers. We do this a lot in C#, and it is compatible with TypeScript design because it won't make any changes in the resulting javascript, and works pretty much like method overloads in TypeScript.
Related discussion: https://typescript.codeplex.com/discussions/452741

Any movement on this one? I've run into the situation where I want to do this:
    export interface Func<R> { (): R; }
    export interface Func<P, R> { (parameter: P): R; }
    export interface Func<P1, P2, R> { (parameter1: P1, parameter2: P2): R; }
    export interface Func<P1, P2, P3, R> { (parameter1: P1, parameter2: P2, parameter3: P3): R; }
    export interface Func<P1, P2, P3, P4, R> { (parameter1: P1, parameter2: P2, parameter3: P3, parameter4: P4): R; }

But Instead I have to define them like this:
    export interface Func0<R> { (): R; }
    export interface Func1<P, R> { (parameter: P): R; }
    export interface Func2<P1, P2, R> { (parameter1: P1, parameter2: P2): R; }
    export interface Func3<P1, P2, P3, R> { (parameter1: P1, parameter2: P2, parameter3: P3): R; }
    export interface Func4<P1, P2, P3, P4, R> { (parameter1: P1, parameter2: P2, parameter3: P3, parameter4: P4): R; }

It's not the end of the world, but it is pretty annoying.

@RonPenton We added generic defaults in #13487 for TS 2.3, so yes - we implemented the equivalent alternative to this.

@weswigham I don't see how defaults are solving the problem with @RonPenton's example
I tried this
export interface Func<R=void> {  (): R; }
export interface Func<R=void, P1=void> {  (param1: P1): R; }
export interface Func<R=void, P1=void, P2=void> {  (param1: P1, param2: P2): R; }

const fn1: Func                          = () => {};
const fn2: Func<string>                  = () => '';
const fn3: Func<string, number>          = (param1: number) => '';
const fn4: Func<string, number, boolean> = (param1: number, param2: boolean) => '';

but the compiler complains at
fn3 with TS2322:Type '(param1: number) => string' is not assignable to type 'Func<string, number, void>'.
and at
fn4 with TS2322:Type '(param1: number, param2: boolean) => string' is not assignable to type 'Func<string, number, boolean>'.
it seem like the compiler does not differ between the second and the third overload of Func
or am I doing something wrong here?

the weird thing is, when I put all overloads in a single interface the compiler accepts fn3 but still complains about  fn4
export interface Func<R=void, P1=void, P2=void> {
  (param1: P1): R;
  (param1: P1, param2: P2): R;
}

const fn1: Func = () => {};
const fn2: Func<string>                  = () => '';
const fn3: Func<string, number>          = (param1: number) => '';
const fn4: Func<string, number, boolean> = (param1: number, param2: boolean) => ''; // error

You should only need to have one interface declaration:
export interface Func<R=void, P1=never, P2=never> {  (param1: P1, param2: P2): R; }

const fn1: Func                          = () => {};
const fn2: Func<string>                  = () => '';
const fn3: Func<string, number>          = (param1) => '';
const fn4: Func<string, number, boolean> = (param1, param2) => '';

The rest are just adding more call signatures, which will get resolved using overload resolution. If you add them; you need to make sure the "most specific" ones are added first, otherwise what you described can happen.

There are some drawbacks with that solution:
All function now must provide 2 parameters.
export interface Func<R=void, P1=never, P2=never> {  (param1: P1, param2: P2): R; }

const fn1: Func                          = () => {};
fn1(); // TS2554:Expected 2 arguments, but got 0.

If I make all parameters optional, then there is no type safety in calling the methods with all expected parameters ....
export interface Func<R=void, P1=void, P2=void> {  (param1?: P1, param2?: P2): R; }

const fn1: Func                          = () => {};
const fn2: Func<string>                  = () => '';
const fn3: Func<string, number>          = (param1) => '';
const fn4: Func<string, number, boolean> = (param1, param2) => '';

fn1(); // fine
fn2(); // fine
fn3(); // should raise error but does not
fn4(); // should raise error but does not

@weswigham I don't believe that solves the issue. I tried to use default generic parameters like @lukas-zech-software did and ran into the same problems. The compiler expects parameters that don't exist.
Try out your code, you'll see that it doesn't work:
export interface Func<R=void, P1=never, P2=never> {  (param1: P1, param2: P2): R; }

const fn1: Func                          = () => {};
const fn2: Func<string>                  = () => '';
const fn3: Func<string, number>          = (param1) => '';
const fn4: Func<string, number, boolean> = (param1, param2) => '';

fn1();                  // Expected 2 arguments, but got 0.
const x = fn2();        // Expected 2 arguments, but got 0.
const y = fn3(123);     // Expected 2 arguments, but got 1.
const z = fn4(1, true); // This is the only one that works.

I don't think default values work for the case I am interested in, either.  I'm trying to get generic parameter binding working.  Creators of a bindable would define the parameters that it accepts, and consumers would bind values for parameters through a "bind" call.  Once all parameters are bound then the bind method is no longer available and the call method becomes available.
This looks something like this
interface IBindable<RequiredParams if-non-empty-object> {
    bind : <ParamSet>(p : ParamSet) => IBindable<RequiredParams less params in ParamSet>;
}
interface IBindable<RequiredParams if-empty-object> {
    call: () => Result;
}

In some ways this is very much like the classic C++ factorial template metaprogramming example


Start Stop Button Not Getting Updated

Bug

Start-Stop Button is not updating whenever we remove them after the process started.

Steps to reproduce the behavior:

Add a file to the process tile
Now start the process by clicking on 'Start All' button.
During the process is going on remove the file in the process tile by clicking on the bin icon.
Now even though the file is removed, the Start-Stop button is not getting updated to 'Start All' from 'Stop All'

Expected behavior

It is expected to change into 'Start All' from 'Stop All' whenever the file is being removed during the process.

Screenshots

Bug

https://user-images.githubusercontent.com/83648898/221263370-b3c31f3a-8cb1-4971-b0a9-d6e2f9a2298d.mp4
Desktop (please complete the following information):**

OS: macOS

this issue will be fixed by pr #48