Vulkan: Transparency pipeline has z-order issues

Godot version
4.0alpha13
System information
Win 10/64 NVIDIA GeForce GTX 1060/PCIe/SSE2
Issue description
Summary of sub issues:

Proximity fade has broken z-ordering
Proximity fade has no shadows #61730
Alpha has broken z-ordering and is overly transparent farther away
Alpha depth prepass is overly transparent which makes it appear that there are  z-ordering issues. Unknown if there's actual  z-ordering inaccuracy
AlphaHash is just noisy with strange artifacts when drawn against the sky.

Alpha scissoring  z-ordering works fine, as documented in #56008. But is not desirable on certain assets like pine trees as looks bad far away.
Proximity Fade
Here is proximity fade on an otherwise opaque object. Has no problem w/ z-ordering in GD3. All the normals are messed up but that's noted in #63550. Both sides in GD4.

Alpha options
On my transparent foliage, I compared the alpha options.
Alpha Scissors
This has proper z-ordering, however it's a low quality alpha. It looks fine close up, but it's ugly from a distance. Compare GD4 alpha scissors up close, far, vs true Alpha in GD3.

Alpha
Alpha has potential. I would use it if these two issues were fixed. Also notice no shadows documented in #61730.

It's too transparent. You can see the grid right through it. Compare with gd3 alpha above. The pine needles look like a ghost tree.

Some faces have the incorrect z-order, showing transparent objects that are behind it, in front. Look at the grid showing through the bottom of the image, too transparent. Another alpha z-order example is coming up below in the depthprepass section, and see the video at the bottom for more egregious examples.

AlphaHash
It's noisy, and has issues with some faces turning very light against the sky. It does not appear to have z-order issues, but I don't know what I'd use this for.

DepthPrePass
There are issues with having too much transparency at times, and z-order at others.  All those light green branches shouldn't be visible here:

Compare with alpha scissors

Overall it doesn't look good. Alpha depth pre pass also didn't look great in GD3, but it was better than alpha or depthprepass in GD4. On the left in GD4, you can see the behind light branches popping in front of the dark front branches. On the right in GD3, you can see the low quality alpha on many of the foreground branches. The alpha really gets chopped up when layered. However the z-order is reasonable. Even in motion, I don't see any background parts obviously rendering in front.

Video
Z-ordering is difficult to see in pictures, so here is video:
https://youtu.be/4_GBr-Pzmk8
Steps to reproduce
n/a
Minimal reproduction project
test_zorder.zip
updated 12/19/22

Turning on gizmos has no effect on these issues. #62457

I considered if this is a duplicate of #54226. It has an appropriate title, but otherwise is a different issue that appears to have been fixed.

Alright, I've made an MRP using some free assets I found, as I can't distribute mine.
test_zorder.zip
Here is the rock with proximity fade. It's messed up everywhere it's not fading in with the ground.  The whole top is messed up.

On the pine tree model, Alpha shows behind faces in front.

But it's not every face. Just random ones. Here 1 & 3 are closest to the camera. Yet 1 properly occludes 2, but 4 incorrectly occludes 3. If I move the camera, 4 also incorrectly occludes 1.

This shot is alpha mode, which is all backwards.

Same shot in Depth prepass, which exhibits some of the issues as on my models. They have too much transparency as the camera pulls back, the grid shows through, and the textures start to thin out too quickly.
On the left the dark textures are getting thin and the light branch shows through easily. On the right, the light textures are thinning and the farther dark branches show through.

Alpha scissors shows proper order

Sources:
https://sketchfab.com/3d-models/rock-free-56592b65d43d4be1b9291fdefb979032
https://sketchfab.com/3d-models/pine-tree-d45218a3fab349e5b1de040f29e7b6f9

I started taking a look at this. For now I have a couple of workarounds until I can figure out what exactly is broken.
For the tree:
One problem with the tree is that all the leaves/branches are in one mesh. So they get drawn in the order specified by the mesh, since transparent objects don't write to depth by default the later branches get drawn over the earlier ones. To fix this some sort of depth prepass is needed (alpha scissor, alpha hash, or depth prepass tested and all work). Also, with TAA enabled, alpha hash looks really good on the tree.
The other problem with the tree is that the leaves look too transparent, you can see the grid through them. This is indeed a z-ordering bug and I don't know exactly what is causing it yet. It goes away with alpha hash and alpha scissor. It should go away with depth prepass too, but its not.
TODO: figure out why grid is consistently drawing over transparent objects and why depth prepass isn't helping
For the rock (proximity fade)
When proximity fade is enabled, the rock essentially turns transparent and even worse, the back faces end up drawing over the front faces. In part this is because back face culling is disabled. But even enabling back face culling doesn't fully solve the issue. This is clearly an issue with how "proximity fade" materials get inserted into the pipeline.
As a workaround for now, the depth-draw mode for the rock can be set to "Always".

As of Beta 8 the issue seems to be resolved for materials within the same mesh, but for whatever reason, in my current project separate meshes are being rendered in the same order regardless of the actual depth. This may also have something to do with the grid always rendering in front of a transparent material.

The alpha of the red cylinder is 0.9, setting it 1.0 removes any ordering issues.
It seems like the z-ordering for transparent materials somehow negates the z-ordering of the individual meshes. I'm not familiar with the Godot source code so I'm hoping someone more knowledgeable than me will be able to fix it using this information.

As of Beta 8 the issue seems to be resolved for materials within the same mesh,

Alpha hash has been significantly improved and now looks like it could be useful for some things. However nothing else has been resolved or improved as far as I can tell.  The only alpha methods with accurate z-order are alpha hash or alpha scissor but both looks terrible for fine details like thin foliage (pine needles) and hair. Scissoring looks fine for thick broad leaves.

As of Beta 8 the issue seems to be resolved for materials within the same mesh,

Alpha hash has been significantly improved and now looks like it could be useful for some things. However nothing else has been resolved as far as I can tell as of 4828ac6b9.
Status of issues:

Proximity fade still turns objects inside out and no shadows
Alpha still has no z-order or shadows
Alpha depth prepass has shadows, but still has artifacts w/ being too transparent and back faces showing in front at some times under certain conditions, eg dark bg light fg or the opposite (which may not be z-order).

The only alpha methods with accurate z-order & shadows all the time are alpha hash (very noisy) or alpha scissor (fine for thick broad leaves but looks terrible for fine details like thin foliage (pine needles), hair and eyelashes. It worked much better in Godot 3.
I have updated the MRP, given the mesh format has changed. The rock shows issues w/ proximity blend turning inside out and no shadows. The tree uses alpha depth-prepass and shows bg faces through the foreground. Look down into the tree and find areas where you see dark areas visible in front of light areas or the opposite. Turn on alpha scissoring and you'll see how incorrect it is.
See the pictures below. Those dark areas especially at the bottom of the image shouldn't be so visible. It's very obvious when the camera is moving. It's also obvious on better quality assets with finer pine needles that we're using but can't share.

Compare w/ alpha scissors

Taking another look at this in Beta 11 and I have a few more observations
The Rock
The biggest difference with the rock in Beta 11 and 3.5.1 is that for some reason in Beta 11 the material imports with backface culling disabled. As a result, you get way more z-order issues than you do in 3.5.1.
3.5.1

Beta 11: default

Beta 11: back face culling enabled

In both cases you need to set the depth draw mode to "always" to get rid of the artifacts (depth draw alpha prepass isn't enough because the alpha value is dependent on the value in the depth buffer and the depth prepass is the pass that creates the depth buffer).
There isn't anything actionable with respect to the rock. Without seeing the original file for your tree trunk file I can't say if there is something else contributing to the problem in that case.
The Tree
I look a closer look at the gizmos drawing through the tree. This is a z-ordering issue came from the  tree being so close to the center of the editor combined with how sorting worked in earlier betas. Since the gizmo's have a massive bounding box they used to be sorted as if they were very far away from the origin (sorting was based on the distance to the nearest face of the bounding box along the camera's direction vector). Now, sorting with a perspective camera sorts based on the distance between the camera origin and the center of the object's bounding box https://github.com/godotengine/godot/pull/69998. Things look much better now and you don't see the editor gizmos through the tree except for where the tree is actually transparent.
I can't see anything actionable with respect to the tree either.
I note you listed concerns in your last comment and I respond to each one here:

Proximity fade still turns objects inside out and no shadows

Like in 3.5.1, depth draw needs to be set to always for proximity fade materials to contribute to the depth buffer and shadows. This should be better documented

Alpha still has no z-order or shadows

This is expected. Transparent meshes draw triangles in the order they are presented as sorting happens on a per-mesh basis. To get proper depth occlusion within a mesh you need to use some sort of depth buffer (depth prepass alpha or depth draw always. In this case depth prepass alpha is the only feasible option. Alternatively you can avoid the transparent pipeline altogether and use alpha scissor or alpha hash). Older games used to pre-sort the triangles in the source mesh to minimize this kind of artifact. For things like trees, they would sort the triangles from top to bottom to get as close to proper z ordering as possible (as trees would normally be view from below). In modern engines we rely more on the depth buffer.

Alpha depth prepass has shadows, but still has artifacts w/ being too transparent and back faces showing in front at some times under certain conditions, eg dark bg light fg or the opposite (which may not be z-order).

This artifact is unfortunate but not something we can avoid. It is especially noticeably on the tree mesh because the needles on the texture are almost never opaque (especially with mipmapping). If you look at the tree using the normal buffer view mode (from the perspective menu) you can see that very little of the needles texture is actually opaque, which means that it suffers from the same sorting issues as the regular Alpha mode.
Ideally for this kind of mesh you would either use alpha scissor with one of the alpha antialiasing modes (AKA alpha to coverage. Although I think this is broken now as the results are not impressive) or you would use alpha hash with TAA.
Unfortunately it seems that both TAA and alpha to coverage need work to be production ready. Hopefully we will be able to fix them soon.

The biggest difference with the rock in Beta 11 and 3.5.1 is that for some reason in Beta 11 the material imports with backface culling disabled. As a result, you get way more z-order issues than you do in 3.5.1.

This is an issue in the glTF importer, as Godot follows the glTF specification and disables backface culling to follow Blender's exported value. We should probably ignore this value and always use backface culling by default (but have an import option to not apply this overrride).

Rock:

When enabling proximityfade, you could force enable backface culling, and depth draw always. Otherwise it is broken without these options, and I suspect there's no documentation that describes these requirements for use.
There are no shadows w/ proximity fade.

Trees:

Alternatively you can avoid the transparent pipeline altogether and use alpha scissor or alpha hash). Older games used to pre-sort the triangles in the source mesh to minimize this kind of artifact.

AS/AH are too coarse for thin foliage and hair. AS looks good up close, but bad far away. Alpha+depth looks good far away but bad up close.

Can't comment on the rock, but I am fairly certain that the tree alpha problems are caused by the mipmaps. My best guess is that godot4 uses a different algorithm to create the mipmaps, which combined with your asset's very thin needles results in too transparent trees when compared to godot3.
There are several solutions to this problem. I personally would switch to sdf alpha, which would give you more control and result in crisp and sharp edges. However, that would obviously require you to touch up all your assets so I can understand if that's not an option for you. Alternatively, you could edit the mips. I know that artists sometimes bring the mips back into PS and scale/remap them there. I would recommend you look into the dds file format, thats what you want to use if you want to have manual control over the mip levels.
Not sure how relevant of a problem this is, but it might also be possible to add an alternative mip creation algorithm to the importer. In theory, it should be trivial to look at the ratio of opaque/transparent pixels in the base level and then automatically modify all the other levels to match that (as closely as possible).
However, I haven't looked at the asset importer in Godot before, so I have no clue how difficult it would be to implement this and whether or not it would be out of scope.

I also have this issue and the two transparent meshes are not even interpenetrating.

I'm making a kind of 2d in 3d type of game and I want textures to have a hand painted feel, which just doesn't look right with alpha hash or raw geometry edges (for foliage especially).
I just want to know if this issue is going to be resolved, as it is crucial to me (I'm in no rush as my camera is mostly fixed and I can hopefully mitigate it for the most part).
And on the side note sharpening is very much needed.
As is textures are painfully blurry. On the right side I'm using FSR at 95% to mitigate it for the time being, but that kills edges, of course. And implementing it on the user side is costly (and motion vectors are not generated for moving textures, so no TAA (though I would prefer SMAA, I'm really hoping it won't be forgotten). Picture clarity is very important, at least to me.

Thank you for bearing with me.

@dame-ng it is hard to tell what the issue is from your screenshots but I don't think it is the same as this issue. For your use case you should look into using sorting_offset

it is hard to tell what the issue is from your screenshots but I don't think it is the same as this issue

Even if it is a different issue I still want interpenetrating geometry to look right. Is that possible? I'm ready to pay the performance cost.
The issue happens when I change the camera to a more acute angle.

For your use case you should look into using sorting_offset

Thank you, I'll look into it.

Enable anisotropic filtering on the materials, and increase the anisotropic filter level to 16×

I already did. It is 16x and material property is set on the screenshot. The bias setting doesn't change anything for the close by textures because it already uses the largest mip, I believe. Bluriness is expected anyway, as GPUs don't do Cubic or Lanczos filtering unfortunately.

If you really want sharpening right now, you can enable FSR 1.0 and set the 3D resolution scale to 99% instead of 95%

I tried both and found that 95% looks basically the same, but the performance doesn't take a hit this way. Edges are ruined anyway.

There's a https://github.com/godotengine/godot/pull/47401, but it needs further work

I'm not ready to plunge into Godot development yet, and I kinda want to just develop games not engines. So I hope someone else would pick it up.

Even if it is a different issue I still want interpenetrating geometry to look right. Is that possible? I'm ready to pay the performance cost.

If sorting_offset doesn't work for your use case, this needs support for order-independent transparency, which is very expensive and not planned for 4.0 (or even 4.1).

@Calinou That's the kind of answer I wanted to hear.
I just find it weird it is such a hard problem, as it was trivial in the past (in the times before programmable shaders), and I always thought the problem was in deferred rendering. So I hoped clustered forward will not suffer from this issue. In a hindsight deferred still used forward rendering for transparency, I guess.
In any way alpha hash won't cut it for me, and going pure 2d has its own issues and limitations that I don't want to deal with. I hope sorting_offset and some camera magic would do the trick.

I have really bad experience working with Transparency/Alpha mode. With this option z ordering not work at all. Take a look on my video to understand what happening. May be I did something wrong, idk.
Uploading 2024-05-29 09-40-39.mp4…

Alpha mode ignores Z ordering. Use alpha depth pre-pass.

When i put 3d sprites with alpha on top of each other, with different positions, but close to each other, depending on the angle I'm viewing the one that should be behind actually appears on top, blocking the vision of the one that should be on top.

To expand further on what @HKunogi said,
The issue is related to Z clipping in 3D space when using Z indices with three decimal places. Specifically, elements with Z values such as 0.001 experience severe clipping, whereas elements with Z values such as 0.01 do not, aligning with the engine's default setting of 0.01. This problem causes elements that are positionally in front of the frame to disappear unexpectedly due to incorrect transparent ordering. Reducing the precision to two decimal places resolves the clipping issue, but this workaround is impractical for scaling our numerical values accurately. The engine should handle Z indices with three decimal places without causing clipping, ensuring that all elements maintain their intended visibility in the 3D space.
This issue and https://github.com/godotengine/godot/issues/43253 as well as https://github.com/godotengine/godot/issues/54226 are related.

The issue is related to Z clipping in 3D space when using Z indices with three decimal places. Specifically, elements with Z values such as 0.001 experience severe clipping,

Can you upload a minimal reproduction project?

Yes,
alphaclip3d.zip
When you open the game and start moving the camera around, you'll notice that elements in the foreground sometimes appear and disappear, even though they should remain visible. This issue persists even at a near clipping plane distance of 0.01, and I’ve tested it thoroughly. It only stops happening at very large sizes, but even at a clipping distance of 1.00, the problem persists. Making the game that large is not feasible for us, as it would introduce new bugs.
@Calinou

@shadowfox87 You might want to try 4.3.beta1 after making a backup of your project, as it comes with reverse Z which greatly improves the usable depth buffer precision.

@shadowfox87 You might want to try 4.3.beta1 after making a backup of your project, as it comes with reverse Z which greatly improves the usable depth buffer precision.

We are already on the latest version of this branch. We compile the source from the master branch. We are not using custom shaders. The reverse Z applies for shaders. The problem remains.

@shadowfox87 In your MRP, it seems setting Render Priority on the sword sprite fixes the sorting issue. You can also use Sorting > Offset in the VisualInstance3D section of the inspector if you need more control.

@shadowfox87 In your MRP, it seems setting Render Priority to 1 on the sword sprite fixes the sorting issue. You can also use Sorting > Offset in the VisualInstance3D section of the inspector if you need more control.
Transparent objects can't Z-fight with other transparent objects, so this is not a depth buffer precision issue (because they don't write to depth in the first place).

It would work on that case, but, lets say that this MRP is a single object and on the scene we would have many of that object (which is a combination of those 3 transparent textures) and they would be close to each other, one in front of the other, in that case, the one with render priority 1 of the object behind, would be rendered on top of the lower priority part of the object that is in front, which shouldn't be the case.

Here, I've attached another M
alphaclip3d.zip
RP where the changing the Rending Priority to 1 will not work. What Kunogi said above is correct.

Hey, just wanted to say that I've been playing with the project.
Let's go back to two comments.

it is hard to tell what the issue is from your screenshots but I don't think it is the same as this issue. For your use case you should look into using sorting_offset
You can also use Sorting > Offset in the VisualInstance3D section of the inspector if you need more control.

I believe Sorting Offset is indeed the correct answer. The main challenge it has, is it is unpredictable unless all sprites have the same position. However, it has a checkbox that allows you to use the AABB instead of the object center.
So, for all the sprites at 0,0,0 the sorting offset will just work. I would suggest small increments like even 0.001, 0.002, 0.003 etc.
For sprites not at 0,0,0 like the swords icon, we can solve it by using the Custom AABB together with the Use AABB Center.

There are three main challenges with using Custom AABB:

You'll need to calculate extents that are big enough to cover the entire rendered area of the sprite, from that center point we are using for sorting. For dynamic Label3Ds, make sure to make it large enough to cover whatever long text string might show up.
Godot's AABB inspector uses origin / extents, not center / extents. This means whenever you change extents, you should use -Extent/2 for the origin point. (the negative corner of the bounding box).
We want the AABB to be centered, so once you have calculated -Extent/2, you need to subtract the node's position relative to the center point. This can get complicated if it is nested a few nodes down.
Here is the scene where I have done this for @shadowfox87's MRP and I can confirm it works correctly if the AABB calculations are done right.
alphaclip3d_issue63675_sorting_order.zip

As for the overarching issue... this issue itself has been a little confused with lots of comments, and it's not asking for anything clearly defined.
I think at least from reading the recent comments re. Label3D and Sprite3D, what people want is a sort of "grouping" node, some sort of mode for in-order drawing of things like sprites, labels, within 3D. This is not a feature Godot has ever had in 3D, but perhaps would be worthy of a proposal if there isn't already one. It would make 3D / 2.5D game programming very familiar to people who have made Godot 2D games in the past, which do use in-order drawing.
The other thing is even for the Sorting Offset approach above, it seems difficult to work with once you have objects with different centers (and need to slide the AABB around). Maybe there would be a way to override that some other way, like specify a Node3D to use as the origin point (and then VisualInstance3D will deal with the complexities around calculating it.)
I think it would be possible to make a script that looks through the tree and calculates the correct sorting offsets and AABBs. If there's interest, I could try and jot down the algorithm describe above into a script that does it on _ready... I don't know how to keep it in sync all the time without a performance cost, but maybe it can have a manual update function that re-calculates the sorting offsets if needed.

As for other proposed solutions...
Rendering Priority works if you have only one or a small number (if you have a script adjust them up and down. not easy). However, I agree that Rendering Priority fundamentally does not work if you want to instantiate more than one: you could adjust all the priorities of everything in the scene to make it work.... until you have more than 32 of them because godot only allows a max of 127 rendering priority.
Viewports would allow you to use the 2D renderer, but then you're in a constant battle with viewport size / resolution and if you have too many large viewports you'll start to hit GPU pixel fill limits especially if you want to support mobile.

@lyuma Thank you for your thorough reply. Yes, you are correct. Although you solved the MRP using custom AABB, we have to recalculate all this for our actual project which is much more complex since it is not native to Godot in 3d. In Unity, this is native so we didn't have to do anything.
As you said, we need a "grouping" mode. Should this require another issue to be opened up to propose this or is it already a feature request given this discussion?

Hey, just wanted to say that I've been playing with the project.
Let's go back to two comments.

it is hard to tell what the issue is from your screenshots but I don't think it is the same as this issue. For your use case you should look into using sorting_offset
You can also use Sorting > Offset in the VisualInstance3D section of the inspector if you need more control.

I believe Sorting Offset is indeed the correct answer. The main challenge it has, is it is unpredictable unless all sprites have the same position. However, it has a checkbox that allows you to use the AABB instead of the object center.
So, for all the sprites at 0,0,0 the sorting offset will just work. I would suggest small increments like even 0.001, 0.002, 0.003 etc. For sprites not at 0,0,0 like the swords icon, we can solve it by using the Custom AABB together with the Use AABB Center.  There are three main challenges with using Custom AABB:

You'll need to calculate extents that are big enough to cover the entire rendered area of the sprite, from that center point we are using for sorting. For dynamic Label3Ds, make sure to make it large enough to cover whatever long text string might show up.
Godot's AABB inspector uses origin / extents, not center / extents. This means whenever you change extents, you should use -Extent/2 for the origin point. (the negative corner of the bounding box).
We want the AABB to be centered, so once you have calculated -Extent/2, you need to subtract the node's position relative to the center point. This can get complicated if it is nested a few nodes down.

Here is the scene where I have done this for @shadowfox87's MRP and I can confirm it works correctly if the AABB calculations are done right. alphaclip3d_issue63675_sorting_order.zip
As for the overarching issue... this issue itself has been a little confused with lots of comments, and it's not asking for anything clearly defined.
I think at least from reading the recent comments re. Label3D and Sprite3D, what people want is a sort of "grouping" node, some sort of mode for in-order drawing of things like sprites, labels, within 3D. This is not a feature Godot has ever had in 3D, but perhaps would be worthy of a proposal if there isn't already one. It would make 3D / 2.5D game programming very familiar to people who have made Godot 2D games in the past, which do use in-order drawing.
The other thing is even for the Sorting Offset approach above, it seems difficult to work with once you have objects with different centers (and need to slide the AABB around). Maybe there would be a way to override that some other way, like specify a Node3D to use as the origin point (and then VisualInstance3D will deal with the complexities around calculating it.)
I think it would be possible to make a script that looks through the tree and calculates the correct sorting offsets and AABBs. If there's interest, I could try and jot down the algorithm describe above into a script that does it on _ready... I don't know how to keep it in sync all the time without a performance cost, but maybe it can have a manual update function that re-calculates the sorting offsets if needed.
As for other proposed solutions...
Rendering Priority works if you have only one or a small number (if you have a script adjust them up and down. not easy). However, I agree that Rendering Priority fundamentally does not work if you want to instantiate more than one: you could adjust all the priorities of everything in the scene to make it work.... until you have more than 32 of them because godot only allows a max of 127 rendering priority.
Viewports would allow you to use the 2D renderer, but then you're in a constant battle with viewport size / resolution and if you have too many large viewports you'll start to hit GPU pixel fill limits especially if you want to support mobile.

Thanks for the in depth response, indeed i could solve some issues with AABB math, by extending it with some calculations, for simple things, but more complex situations, like the ones described can't be solved properly when taking into account performance and scalability for the objects on a dynamic game. What you said is correct, in that, having a grouping object, that ensures that objects inside that group are drawn in the order of the index they are on their parent, would solve the issue, unity does that actually, if i have a parent object X, and it has children Y and Z, with Y being index 0 and Z index 1, it will draw the Y and then afterwards the Z on top of it, but from my tests, godot does not do that (at least on 3D?), having somehow make it do that, would definitely solve that Z sorting issues for all cases where they are handled by the game maker.

Running into Z order issues myself, and I set up a simple project to try to understand what was going on before finally running into this thread.

Godot Z Depth Transparency Shader Test.zip

This provided project just consists of created planes, a couple cubes, and a suzanne imported from Blender. All meshes just have a material applied to them that uses an incredibly simple shader that passes the desired color onto the ALBEDO and ALPHA channels in the fragment shader.
Interestingly, the red plane at 0,0,0 has no z depth problems that I see
Raising the dark plane to make its distance further from the below planes mitigates, but does not solve, the sorting issue
This test is consistent with Godot 4.2.2, godot-master (b75f048), and persists on a recreated project on Godot 3.5.1 (which I simply already had installed on a laptop)

@champbob You are running into the standard difficulty with transparency sorting. What you see isn't a bug, but is instead a general limitation with real time 3D rendering.
Image from OpenGL wiki https://www.khronos.org/opengl/wiki/Transparency_Sorting

Take the above example, If A is the camera, what object should be drawn in front, B or C? Intuitively, it should be C. However, when sorting the two objects, B will come out in front because the center of B is closer to the camera than the center of C. This is what is happening in your example. You will run into this problem whenever you have large transparent objects near other transparent objects that require depth sorting. This is why you rarely see large semi-transparent surfaces in video games.
We have some tools to help deal with this and they are explained in detail in the docs.
https://docs.godotengine.org/en/latest/tutorials/3d/3d_rendering_limitations.html#transparency-sorting

@clayjohn Ah, thanks for the explanation! Apologies for misinterpreting my issue!

@clayjohn
I have a scene with two quads, slightly offset.
Both quads have a material with these options set:
transparency = Depth Pre-Pass
shading_mode = Unshaded
On Forward+, there is no bug.
On Mobile, when viewed where the center of the quad in the back is closer to the camera, the quad in the back renders on top of the quad in front (see screenshots below).  This is likely due to the explanation you gave above, though I would've expected this for "transparency = Alpha", not for "transparency = Depth Pre-Pass".

Is "'transparency = Depth Pre-Pass' is broken on Mobile" a separate bug?  I couldn't find any related documentation.

Is "'transparency = Depth Pre-Pass' is broken on Mobile" a separate bug? I couldn't find any related documentation.

The Mobile rendering method doesn't use a depth prepass, so it can't work there.

@dhoverml You are actually running into a different bug. Its already reported here: https://github.com/godotengine/godot/issues/92292. The depth-prepass transparency option is broken in the Mobile and Compatibility backends.

The Mobile rendering method doesn't use a depth prepass for performance reasons, so it can't work there.

Ah, good to know that it is expected.  Maybe there should be a warning in the editor when Depth Pre-Pass is chosen on Mobile?
For example, if I enable subsurface scattering on Mobile I see this:

servers/rendering/renderer_rd/forward_mobile/scene_shader_forward_mobile.cpp:165 - Sub-surface scattering is only available when using the Forward+ rendering backend. (User)


Update service-fabric-reverseproxy.md

Update to apiVersion "2017-07-01-preview"
Update json values to use typed values (boolean, int) instead of strings

Ping. If you guys don't intend to merge this, please close the PR.

@sridmad can you review this change?


Configuration on distributed mode

Hi,
I've read this link but I can't figure out how to do the load balancing.
I have two orientdb servers (started with ./dserver.sh) (2424 and 2425) running on my machine (just for testing purposes).
In the client, each database connection comes form:
url = "remote:localhost/Demo";
pool = new OPartitionedDatabasePool(url, user, password);
...
pool.acquire();

But, for what I understand, this will only query against 2424.
How can I configure the load balancing?
Thank you,

That link's title is misleading, instead of load balancing it really only explains (and should be re-titled) how to setup automatic client fail-over from one server to another using a DNS trick.
In your case orient's recommendation is usually Sharding.  You might also use a software load-balancer like Zen Load Balancer or HAProxy

Hi @8forty.
Can you provide an example for my case using orient's recommendation?
Thanks,

I don't have an example using sharding, we decided for our project to simply use multiple identical orient servers on multiple machines, each ingesting all of the data.  This provides high-availability and fault-tolerance, and for load balancing our clients just randomly choose which one to connect to.

Thanks @8forty. I see you opened an issue (#3575). It would be great to have more documentation about this.
If you have n machines, that means you have n pools (like my example above), and the client chooses a pool randomly?
It's a good load balancing? Since on machine might be overloaded and still be chosed.

Yeah, it's not really a very good load balancing scheme in general but works well in our use-case since we are almost never overloaded by clients.  I think HAProxy (for example) has features that let it monitor the load of the servers it's balancing and choose the one with the lightest load.

Thanks again! :+1:

Duplicate of: #3165


Add converters

We have validators to allow reattempting to get correct API output. It would also be nice to have converters. If a converter throws an exception we can also catch and retry retries times.

@mentat-butler can you attempt this? Add a test that shows using a converter to parse to a pydantic object


Improve version handling in the docs

This PR will help eliminate a lot of manual file editing for the Docs team.
Fixes DOC-894, DOC-2263

This change is

Need to make more adjustments.

@nickvigilante, you're referencing https://cockroachlabs.atlassian.net/browse/DOC-2263, but I don't see how it factors in. Can you help me understand?

@nickvigilante, you're referencing https://cockroachlabs.atlassian.net/browse/DOC-2263, but I don't see how it factors in. Can you help me understand?

See this file: https://github.com/cockroachdb/docs/blob/7eb7a10c19dcde60cc5164631173d2ad4d16a2bc/_data/versions.csv
Notice how we now track the maintenance support expiration date and assistance support expiration date.

Nice! What are the implications for release_info in _config_base.yml?
I think I was also expecting the EOL dates to somewhat be reflected on the release index page as well. But that's not what was in focus. Something for us to consider in the future, if it even make sense.
A lot of great work here, @nickvigilante!

Nice! What are the implications for release_info in _config_base.yml?

The hope is that we will eventually retire making frequent changes to _config_base.yml. The next phase of my changes is to move the existing Liquid blocks to use what's in the _data folder instead. Everything should still display as expected. I included a few examples in this PR (v21.2/secure-a-cluster.md).

I can also add those expirations to the release index page. After I merge this, I'll send another PR.

@lnhsingh Understood. I'll start up a draft page and share it.


Added support for TypoScript

Added support for TypoScript/TSConfig highlighting.

1. None of the regex literals need a `g` flag. If Prism needs it, it will add it.

Will be fixed! ;)

2. We use tabs for indentation.

Will check this too.

3. Please add some [tests](https://prismjs.com/test-suite.html).

I take a short look at it, but to be honest, i already spend way to much time on this language creation. 😉

Two further questions:

Are comments allowed, explaining some of the regex in the non-minified file?

Is there a way to use internal reverence, or is it possible to declare a variable somewhere to store something? I need the keywords matching regex on two places. To save data, i could use the same regex for both. Is there any way or should i just insert them two times?

Are comments allowed

Of course!

Is there a way to use internal reference, or is it possible to declare a variable somewhere to store something? I need the keywords matching regex on two places. To save data, i could use the same regex for both. Is there any way or should i just insert them two times?

Here is how the Java language definition does it. Please note that the IIFE is necessary to prevent namespace pollution.

Thank you for your reply. I'm now completely finished with my work, but the default tests fails, because of one regex group. But i have no idea how to fix this. I need a bit of help here.
The Problem is my comment matching. General a comment always starts with //. But if there is a leading : or \\ it is not a comment, or whenever there is = with optional space directly in front, it's not a comment too. I have the regex working fine, but the prism tests fails, because there is a matching group. But i can't make the group non-matching with ?:, because of the lookbehind.
My regex:
/(^|(?<!\\|:|=\s*))\/\/.*/

Here is a working example of it. Everything is fine, but i would not pass the test:
https://regex101.com/r/RYh0Og/1/
Would be nice if someone has a hint how to do this. Thanks!

The problem in /(^|(?<!\\|:|=\s*))\/\/.*/ is the ES2018 lookbehind (?<!\\|:|=\s*). ES2018 lookbehinds are fairly new, so we can't use them. That's why we have these pseudo-lookbehind groups via a capturing group.
But the problem with Prism lookbehind groups is that we can't easily negate them. This is not trivial, so here's your pattern with the ES2018 lookbehind: /(^|[^\\:=\s]|(?:^|[^=\s])\s+)\/\/.*/
However, your tests still don't pass because with the second comment here not being highlighted:
none match text // a comment
// a comment

Why is the second comment not highlighted? The problem is that after the first comment, the next character is a \n. \s matches \n, so we need to find a character that matches [^\s=] before it. But we cannot find such a character because the regex engine started matching at \n and cannot go back (without ES2018 lookbehinds) to the t of the previous comment. Thus the second comment cannot be matched.
How do we fix that?
We can't... We can only change the pattern. If we use the pseudo-lookbehind of (?<!\\|:|=[ \t]*) instead, it will work. This means that code like the following to be a false positive but I hope that's acceptable.
=
// false positive

Final pattern: /(^|[^\\:= \t]|(?:^|[^= \t])[ \t]+)\/\/.*/

Thank you @RunDevelopment ! I've tested your pattern and it works perfectly! Such things are way above my regex-knowledge. ;)
So, a second before your comment, I've pushed my changes. I've completely redone my language definition and managed to removed the whole tag list, so it is now more a syntax based highlighting. I'm pretty pleased with the outcome. The highlighting is much better then in my first submission. Overall, i have noting left to do. Everything is covered so far.
I think i have managed to get all your things in the review covered. I've added some tests too and hope that everything is ok now.

I think I've resolved everything so far. Hopefully we're fine now. 😉

@dkern The last remaining issue is the comment pattern. The easiest solution might be to copy it from clike.

@dkern The last remaining issue is the comment pattern. The easiest solution might be to copy it from clike.
Done.

Thank you for contributing @dkern!


Allow users to easily set up Ads via Ad Manager

Feature Description

We already have people doing this manually using other plugins (sample support thread). Allow for something similar in the settings panel of the plugin.
Alternatives Considered

Additional Context

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Implementation Brief

Assigning WP pod too because this will require PHP changes

And, per your comment on #4803 these two efforts are related (AdSense and Ad Manager support) so I'll merge into one ticket.


shouldShowRationale does not working properly

Description
This is the code that I wrote for handling permission:
request.addListener {
    when {
        it.allGranted() -> onPermissionAccepted()
        it.anyShouldShowRationale() -> showRationaleDialog() { request.send() }
        it.anyPermanentlyDenied() -> showNeverAskAgainDialog()
        it.anyDenied() -> onPermissionDenied?.invoke()
    }
}

When I call request.send() for the first time and after that I press Deny button, inside the listener I get shouldShowRationale() as true and because of that immediately my  RationaleDialog shows up. And even If I dismiss that dialog and call request.send() again, instead of getting anyShouldShowRationale as true (to show RationaleDialog before the system permission dialog) it shows the system permission dialog. Is this an expected behavior or am I missing something?

KPermissions version:
3.2.1
API level:
All
Expected Behavior:

User clicks on a button so request.send() gets called
System permission dialog shows up
If user press on Deny button the listener should not give anyShouldShowRationale() as true and it must return anyDenied() as true
After user presses the same button again, We call request.send() again
This time before system permission dialog. the listener must return anyShouldShowRationale() as true so that we can show rationale dialog and after user clicks on OK button inside that dialog we have to show system dialog.

This is the expected Android behavior.
To be more precise:

If user press on Deny button the listener should not give anyShouldShowRationale() as true and it must return anyDenied() as true

When anyShouldShowRationale is true it means that anyDenied is true, but, if anyDenied is true, it does not mean necessarily that anyShouldShowRationale is true.
To be clearer: the only case in which anyDenied is true and anyShouldShowRationale is false is when a permission is denied permanently (the user clicked on "never ask again").
You can still implement the logic you described doing the checks before sending the permission request (you can use PermissionRequest.checkStatus()).
Feel free to re-open the issue if something is unclear.


[🔥] Removed Settings override

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Document readonly/IEnumerable navigation patterns.

Hello,
I want to ask about collection in EF Core which are used to relations. For example I have entity Invoice and InvoiceItem. Invoice implements public ICollection<InvoiceItem> Items {get; set;} property.
But ICollection exposes Add method so client can call invoice.Items.Add(new InvoiceItem(...)); It is violation of domain model rule that only aggregate root can manage entities inside aggregate.
When I change implementation to IEnumerable(IReadOnlyCollection)<InvoiceItem> there is not Add method so client cannot add new InvoiceItem to Invoice. But Invoice entity cannot too. There would have to be Add method on Invoice:
public void AddItem(int price)
{
    var oldItems = new List<InvoiceItem>(Items);
    oldItems.Add(new InvoiceItem(price));
    Items = oldItems.;
}

Question is how EF tracker will be worked that collection is new object.
I can add private field private List<InvoiceItem> items = new List<InvoiceItems>(); Then I change public IEnumerable<InvoiceItem> Items => items;
There are two question:

How to map backing field for it?
How will work Include method ... dbContext.Invoices.Include(k=>Items)?

Second problem is that I have public setter on InvoiceItem. Can EF work with private set method on navigation property?
Is there any possibility how to work with collections to satisfy domain model rule that entities in aggregate can manage only root entity?
Thank you.

@ajcvickers Thank you very much. This is exactly what I needed to know :)


Fixed displaying of months for 'RU' and other locales

@g-gray Thanks so much for your contribution, and sorry for the delay.


Make progress bars constant width

Before:
     Running `target\debug\probe-rs.exe run C:\_Hobby\rpi-smoke-tester-setup\embedded-test\esp-hal\output\esp32s2`
      Erasing ✔ [00:00:01] [########################################] 192.00 KiB/192.00 KiB @ 189.88 KiB/s (eta 0s )
  Programming ✔ [00:00:02] [############################################] 27.95 KiB/27.95 KiB @ 9.52 KiB/s (eta 0s )
    Finished in 2.9602304s

After:
     Running `target\debug\probe-rs.exe run C:\_Hobby\rpi-smoke-tester-setup\embedded-test\esp-hal\output\esp32s2`
      Erasing ✔ [00:00:01] [###################################] 192.00 KiB/192.00 KiB @ 189.35 KiB/s (ETA 00:00:00)
  Programming ✔ [00:00:02] [###################################]  27.95 KiB/ 27.95 KiB @   9.53 KiB/s (ETA 00:00:00)
     Finished in 2.96s

Changed the display format of ETA, and Finished
Aligned Finished with the other labels
Widened the current/total bytes and throughput fields

Closes #2736

Yeah this barely fits on 80 characters but not sure what to do about it, except hide something

Current iteration: fixed (<80) width. I've opted to display ETA while a progress bar is running, and replace it with the elapsed time once done - the jump may look weird if a stage is quick, but there is some sense in the choice I hope.
       Unfortunately what I tried looked pretty bad 🫠


when we click generate source. nothing happens

when we click generate source. nothing happens

try reinstall without any  av

To reinstall what? I have the same issue

because generate source does only backend stuff and it's not displaying anything special


fix: completed 8021x credential context typing

:tada: This PR is included in version 2.5.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


DOC: outline workaround when precision losses occur in BFGS [skip cir…

This PR outlines what to do when result.success is False and result.message == 'Desired error not necessarily achieved due to precision loss' when using BFGS. This was reported in #19024. The reason this happens is the gtol termination criterion can be too small, so the minimisation loop never terminates. Eventually the linesearch reaches a point where it can't go any further because it recognises the precision in numerical differentiation means further iteration isn't possible, hence the reported message.
The solution is to relax the gtol convergence parameter.
For problems in double precision the default value of gtol=1e-5 seems to work well.
For problems in single precision gtol may need to be relaxed to e.g. 1e-3.

@jakevdp @paulestano

If you know where gtol comparison is done, we can make it problem size aware. Seems like the typical 100*max(m,n)*(s,d)lamch is enough. (single eps is ~1.2e-7, double is ~2.2e-16)

I'm not sure to understand the intuition behind your tol choice @ilayn. However, I think it's important to keep the users aware of the actual distance to the optimum and let them decide of the tolerance

If you know where gtol comparison is done, we can make it problem size aware

The vector norm of g is calculatedhere. This is compared to gtol here. For large problem sizes the vecnorm will increase in size, so some sort of scaling for problem size makes sense.


travis: test osbuild itself in travis

This adds two tests: noop and yum.
Noop just runs osbuild with one noop stage and a noop assembler.
Yum creates a Fedora 27 build image containing dnf and systemd. F27 was chosen as it is the most recent version of Fedora that can installed using yum. The reason to use yum at all is that travis runs on Ubuntu and ubuntu shipps yum (but not dnf).

Awesome work!

Just the invocation (arguments are slightly different in uninteresting ways (because, of course they are)):

They are literally the same, except for install_weak_deps, which we probably want for yum as well? What do you think of merging them into one stage?

Weak deps are not a thing in yum. The dnf syntax does not work in yum (need to use "key=value" rather than "key", "value"), but it may be possible the other way around. I think also the repodirs thing is not needed in yum (and currently causes a warning).
My thoughts here was that rather than merging them completely, maybe pull out some helpers (the config file format is the same for instance). What do you think?

Weak deps are not a thing in yum. The dnf syntax does not work in yum (need to use "key=value" rather than "key", "value"), but it may be possible the other way around. I think also the repodirs thing is not needed in yum (and currently causes a warning).

Oh wow, I wouldn't have expected that. Don't worry about it then. It's a small stage and not a lot of duplication.

My thoughts here was that rather than merging them completely, maybe pull out some helpers (the config file format is the same for instance). What do you think?

Meh :) It's really not that much, so I don't think we should worry. I just didn't want slighty-different stages for no reason.


$invalid/$valid should only be evaluated when $dirty

Basically do not determine if the for or any of the fields are invalid until the form is dirty.

Only required fields are immediately validated and set $valid/$invalid to prevent submission of data without required fields. This behavior matches Angular forms.

required fields should not be validated until input has happened... this causes all sorts of issues with the message displaying immediately on page load, really bad UI.


how can I

I want to get a result picture after softpooling to compare with other method,how can I do ?

Hi @zhangxue2115 ,
You can convert your images to a pytorch tensor and then use the corresponding soft_pool2d() function

I am very happy to receive your reply, but I still have some questions. After my picture is converted into a pytorch tensor, torch Size =([3, 1080, 1920]), when I use soft_ Pool2d, it will report a mistake atx = F.avg_pool2d(x.mul(e_x), kernel_size, stride=stride).mul_(sum(kernel_size)).div_(F.avg_pool2d(e_x, kernel_size, stride=stride).mul_(sum(kernel_size))),showing"Given` input size: (3x1x1920). Calculated output size: (3x-4x956). Output size is too small" where is my error? I cant find it...

Your reply is very helpful to me.Thank you for your patience.


Maximum Line Length

The Code Style section states:

The Python code in this repo is meant to largely obey the PEP8 style guide.

PEP8  states:

Limit all lines to a maximum of 79 characters.

However we encourage developers to check code style with:
flake8 [your-code-here.py] --max-line-length=99 --select=E,W

Why do we use a max-line-length of 99 characters instead of 79 characters?

https://github.com/exercism/xpython/pull/299

I like this number because it is prescribed by the style guide we conform to. I don't see a reason to override this standard.

We only "largely obey the PEP8" ... and I just see no reason to work with a hard limit of 80 characters only because IBM choose that number for their punchcards. The goal should be to avoid long lines but not at all costs.

I didn't know the IBM punchcard was the origin for the 80 character limit - thanks for sharing!
This StackOverflow answer resonates with me:

To view multiple source windows side-by-side or using a side-by-side diff viewer.

Also the Google Style Guide also uses a 80 character limit.

Most of the arguments for the limit of 80 are totally esoteric ... but just working with flake8 default settings would make things a little bit easier for new people. But to get to that point we would need to uglify some parts of our code like grep_test.py only because of one or two characters over the hard limit:
    def test_multiple_files_several_matches_print_line_numbers_flag(self):
        self.assertMultiLineEqual(
            grep("that", ["iliad.txt", "midsummer-night.txt", "paradise-lost.txt"], "-n"),
            ("midsummer-night.txt:5:But I beseech your grace that I may know\n"
             "midsummer-night.txt:6:The worst that may befall me in this case,\n"
             "paradise-lost.txt:2:Of that Forbidden Tree, whose mortal tast\n"
             "paradise-lost.txt:6:Sing Heav'nly Muse, that on the secret top\n")
        )

    def test_multiple_files_several_matches_case_insensitive_flag(self):
        self.assertMultiLineEqual(
            grep("TO", ["iliad.txt", "midsummer-night.txt", "paradise-lost.txt"], "-i"),
            ("iliad.txt:Caused to Achaia's host, sent many a soul\n"
             "iliad.txt:Illustrious into Ades premature,\n"
             "iliad.txt:And Heroes gave (so stood the will of Jove)\n"
             "iliad.txt:To dogs and to all ravening fowls a prey,\n"
             "midsummer-night.txt:I do entreat your grace to pardon me.\n"
             "midsummer-night.txt:In such a presence here to plead my thoughts;\n"
             "midsummer-night.txt:If I refuse to wed Demetrius.\n"
             "paradise-lost.txt:Brought Death into the World, and all our woe,\n"
             "paradise-lost.txt:Restore us, and regain the blissful Seat,\n"
             "paradise-lost.txt:Sing Heav'nly Muse, that on the secret top\n")
        )

    def test_multiple_files_several_matches_print_line_numbers_flag(self):
        self.assertMultiLineEqual(
            grep("that",
                 ["iliad.txt", "midsummer-night.txt", "paradise-lost.txt"],
                 "-n"),
            ("midsummer-night.txt:5:But I beseech your grace that I may know\n"
             "midsummer-night.txt:6:The worst that may befall me in this case,"
             "\nparadise-lost.txt:2:Of that Forbidden Tree, whose mortal tast"
             "\nparadise-lost.txt:6:Sing Heav'nly Muse, that on the secret top"
             "\n"
             ))

    def test_multiple_files_several_matches_case_insensitive_flag(self):
        self.assertMultiLineEqual(
            grep("TO",
                 ["iliad.txt", "midsummer-night.txt", "paradise-lost.txt"],
                 "-i"),
            ("iliad.txt:Caused to Achaia's host, sent many a soul\n"
             "iliad.txt:Illustrious into Ades premature,\n"
             "iliad.txt:And Heroes gave (so stood the will of Jove)\n"
             "iliad.txt:To dogs and to all ravening fowls a prey,\n"
             "midsummer-night.txt:I do entreat your grace to pardon me.\n"
             "midsummer-night.txt:In such a presence here to plead my thoughts"
             ";\nmidsummer-night.txt:If I refuse to wed Demetrius.\n"
             "paradise-lost.txt:Brought Death into the World, and all our woe,"
             "\nparadise-lost.txt:Restore us, and regain the blissful Seat,\n"
             "paradise-lost.txt:Sing Heav'nly Muse, that on the secret top\n"))

@rootulp I really like your input and this track could use more people like you. So let's just merge #438 and get real things done instead of arguing about tabs vs spaces or 79 vs 99 max line length.

but just working with flake8 default settings would make things a little bit easier for new people

Awesome! I feel strongly that this is the case.


Implement hpack-hash feature.

This implements the feature request in issue #380.
I used the name hpack-hash to make it clear that it is a hpack feature, not a cabal one.
I am happy to change the implementation and attribute name as needed.

Could you make it also suppress the hpack version, and anything that causes vcs conflicts. Cf #380

Thanks so much for this! I've been pulling my hairs for hours why i couldn't build a specific stack.lock.yaml file anymore only to realise that my OS ships with a different version of hpack than stack ships with by default https://github.com/NixOS/nixpkgs/issues/84820#issue-597327259
Not having the Hpack version in the generated cabal file seems to exactly solve that problem

@simfleischman thanks for working on this!

I agree that hashes in generated cabal files are a pain.  In the past, there were sporadic complaints about stack overwriting modified cabal files, this is why @snoyberg  eventually proposed modification detection by using hashes.  This made sense at the time, but in retrospect I'm puzzled whether it hurt more people than it helped.

From my perspective package.yaml is a definition for a Haskell package.  With that perspective, I'm not sure if adding hpack-hash to package.yaml is a good fit.  I opened #390 as an alternative proposal. Everybody who cares about this, please try it and make sure that it solves your use case.

I understand that the hpack version can lead to friction, but it also serves a purpose.  Let's treat this as a separate story.

#390 meets this need


triclinic system

Haven't checked if it works for triclinic system

fixed with the right equation to calculation a*


Inconsistency between keras model.predict() and model.call()

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.1
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v2.0.0-rc0-101-gd2d2566 2.0.0-rc1
Python version: Python 3.7.4
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
I am trying to extract the features generated by the model's DenseFeatures layer by creating a new model based on the inputs of the original model and the outputs of the DenseFeature layers. When using this submodel I notice an inconsistency between model.call() and model.predict(). If we provide extra columns to the model in model.predict() the model behaves reliably by processing the inputs using keras.engine.training_utils.standardize_input_data(). However if model.call() is used this doesn't happen and the model orders the columns based on alphabetical order as per nest.flatten(), thus the model crashes with the type cast exception if the extra columns are of a different format. Please see the code provided to reproduce this issue. I can work around this by using the predict() method.
Describe the expected behavior
Expected behaviour is that the two methods to use a model are consistent. The model.call() processes the inputs using the standardize_input_data() method to ensure the input data is as expected.
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
import urllib.request as request
import tensorflow as tf
import pandas as pd

def download_data(download_path: str):
    url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data'
    header_line = 'age,sex,cp,trestbps,chol,fbs,restecg,thalach,exang,oldpeak,slope,ca,thal,target\n'

    # Download the data and add the column headers
    request.urlretrieve(url, 'heart.data')
    with open(download_path, 'w') as output:
        output.write(header_line)
        with open('heart.data', 'r') as input_data:
            output.writelines(input_data.readlines())

def preprocess_df(df, categorical_columns):
    """Ensure categorical columns are treated as string inputs"""
    col_types = {key : str for key in categorical_columns.keys()}
    df = df.astype(col_types)
    return df

def df_to_dataset(dataframe, target_column='target', shuffle=True, batch_size=5):
    """Dataset preparation code from the tensorflow tutorial"""
    dataframe = dataframe.copy()
    labels = dataframe.pop(target_column)
    ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), tf.one_hot(labels, depth=2)))
    if shuffle:
        ds = ds.shuffle(buffer_size=len(dataframe))
    ds = ds.batch(batch_size)
    return ds

if __name__ == '__main__':

    # Download dataset
    data_path = 'heart.csv'
    download_data(data_path)
    df = pd.read_csv(data_path)

    # Setup feature columns
    numeric_columns = ["age", "chol"]
    categorical_columns = {"thal": df['thal'].unique()}
    feature_columns = {}
    inputs = {}
    for feature_name in numeric_columns:
        feature_columns[feature_name] = tf.feature_column.numeric_column(feature_name)
        inputs[feature_name] = tf.keras.Input(name=feature_name, shape=(), dtype=tf.float32)

    for feature_name, vocab in categorical_columns.items():
        vocab.sort()
        cat_col = tf.feature_column.categorical_column_with_vocabulary_list(feature_name, vocab)
        feature_columns[feature_name] = tf.feature_column.indicator_column(cat_col)
        inputs[feature_name] = tf.keras.Input(name=feature_name, shape=(), dtype=tf.string)

    # Prepare input data
    df = preprocess_df(df, categorical_columns)
    batch_size = 5  # A small batch sized is used for demonstration purposes
    train_ds = df_to_dataset(df, target_column='target', batch_size=batch_size)

    # Create Model
    input_tensors = []
    feature_names = list(feature_columns.keys())
    feature_names.sort()
    for column_name in feature_names:
        features = feature_columns[column_name]
        x = tf.keras.layers.DenseFeatures(features, name=f'{column_name}_feature')(inputs)
        input_tensors.append(x)

    x = tf.keras.layers.Concatenate()(input_tensors)
    x = tf.keras.layers.Dense(units=24, activation='relu', name='dense_0')(x)
    x = tf.keras.layers.Dense(units=24, activation='relu', name='dense_1')(x)
    y_pred = tf.keras.layers.Dense(units=2, activation='softmax', name='output_layer')(x)
    model = tf.keras.Model(inputs=inputs, outputs=y_pred)

    model.compile(optimizer=tf.keras.optimizers.Adam(),
                  loss=tf.keras.losses.CategoricalCrossentropy(),
                  metrics=['accuracy'],
                  run_eagerly=True)

    model.summary()
    model.fit(train_ds, epochs=1)

    # Create a new keras model to extract the features
    # the actual model is using.
    outputs = []
    for column_name in feature_names:
        outputs.append(model.get_layer(f'{column_name}_feature').output)

    feature_extractor = tf.keras.Model(model.input, outputs)

    for i, (X, _) in enumerate(train_ds):

        # Predict works as it calls keras.engine.training_utils.standardize_input_data() internally
        # this modifies the input so that if extra columns are passed they are removed and column
        # order is changed as per the model inputs specified.
        # out = feature_extractor.predict(X)

        # Model call() doesn't use the above util method and thus fails
        # as the ordering of the input columns doesn't match the input
        out = feature_extractor(X)
        print(out)

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Traceback (most recent call last):
  File "check_model_clone_4.py", line 103, in <module>
    out = feature_extractor(X)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 707, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 859, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/dense_features.py", line 133, in call
    self._state_manager)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py", line 4357, in get_dense_tensor
    return transformation_cache.get(self, state_manager)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py", line 2608, in get
    transformed = column.transform_feature(self, state_manager)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py", line 4296, in transform_feature
    transformation_cache, state_manager)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py", line 3771, in get_sparse_tensors
    transformation_cache.get(self, state_manager), None)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py", line 2608, in get
    transformed = column.transform_feature(self, state_manager)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py", line 3749, in transform_feature
    return self._transform_input_tensor(input_tensor, state_manager)
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py", line 3726, in _transform_input_tensor
    prefix='column_name: {} input_tensor'.format(self.key))
  File "/home/user/anaconda3/envs/rc1/lib/python3.7/site-packages/tensorflow_core/python/feature_column/utils.py", line 58, in assert_string_or_int
    '{} dtype must be string or integer. dtype: {}.'.format(prefix, dtype))
ValueError: column_name: thal input_tensor dtype must be string or integer. dtype: <dtype: 'float32'>.

I was able to replicate the issue for TF-2.0rc1 and TF-2.0nightly. Kindly find the gist of colab.thanks!

Here is a short, self-contained example where call and predict disagree (without raising an exception).
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
print("tensorflow version",tf.__version__)

# make a null base model
baseinputs = keras.Input(shape=(1,), name='base_input')
base = keras.Model(inputs=baseinputs, outputs=baseinputs)

# Use it to construct a bigger model.
# (I now know there is a better way to make this model.)
Nin,Nout,Nparallel = 1,1,2
parallel_inputs = keras.Input(shape=(Nin,Nparallel), name='parallel_input0')
# apply base NN to each parallel slice; each outputs (?,Nout)
xs = [base(layers.Lambda(lambda x : x[:,:,i],name='view'+str(i))(parallel_inputs)) for i in range(Nparallel)]
# reshape each of them to (?,Nout,1)
xs = [layers.Reshape((Nout,1))(x) for x in xs]
# concatenate on the third direction to get (?,Nout,Nparallel)
cx = layers.Concatenate()(xs)
# create input scalars for weighted sun (?,Nparallel)
weight_inputs = keras.Input(shape=(Nparallel,), name='parallelScalars')
# do a weighted sum over the third direction to get (?,Nout)
out = layers.Dot((-1,-1))([cx,weight_inputs])
wrapper = keras.Model(inputs=[weight_inputs,parallel_inputs], outputs=out, name='parallelwrapper')

# Make tiny example and try predict and call
w = np.array([[7.,11.]],dtype='float32')
v = np.array([[[3.,5.]]],dtype='float32')
wrappercall = wrapper([w,v]).numpy()
wrapperpredict = wrapper.predict([w,v])
print("wrapper ||call - predict|| =",np.linalg.norm(wrappercall-wrapperpredict))
print("wrapper predict =",wrapperpredict,"; correct is 3*7+5*11=",3*7+5*11)
print("wrapper call =",wrappercall,"; appears to do 5*(7+11)=",5*(7+11))

"""
Outputs:

/usr/lib/python3/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
tensorflow version 2.0.0
2019-10-21 14:27:01.188059: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-21 14:27:01.211364: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-10-21 14:27:01.211915: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54e1d50 executing computations on platform Host. Devices:
2019-10-21 14:27:01.211940: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
wrapper ||call - predict|| = 14.0
wrapper predict = [[76.]] ; correct is 3*7+5*11= 76
wrapper call = [[90.]] ; appears to do 5*(7+11)= 90

"""

same issue here.
@tanzhenyu @jvishnuvardhan any updates?

tf1.14.0 can reproduce the bug, both eager mode and graph mode.

Gently ping @qlzh727 @rchao @pavithrasv; does this looks familiar to you?

@byronyi  @MartinMohlenkamp  using the standalone repro code to depict the difference between call and predict, I am unable to reproduce this with 2.2.0-rc1.
@nikhil-mathew-td can you please try with 2.2.0-rc1  and close this if the issue is fixed for you.

@nikhil-mathew-td @MartinMohlenkamp Thanks for the report! (And sorry for the late reply)
Looks like this is fixed in tf-nightly. Would you mind confirming it before I close the issue?

@tanzhenyu the original code to reproduce with DenseFeatures still fails but at a new location. Now the code fails at model.fit() with the following error:
2020-04-01 11:56:24.474889: W tensorflow/core/framework/op_kernel.cc:1706] OP_REQUIRES failed at cast_op.cc:124 : Unimplemented: Cast string to float is not supported
Traceback (most recent call last):
  File "/home/nikhil/workspace/testing_ground/test_tf_bug/check.py", line 86, in <module>
    model.fit(train_ds, epochs=1)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 71, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 920, in fit
    tmp_logs = train_function(iterator)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 608, in __call__
    result = self._call(*args, **kwds)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in _call
    return self._stateless_fn(*args, **kwds)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnimplementedError:  Cast string to float is not supported
	 [[node model/Cast_1 (defined at home/nikhil/workspace/testing_ground/test_tf_bug/check.py:86) ]] [Op:__inference_train_function_941]

Function call stack:
train_function

The code goes through if the  tf.data.Dataset.from_tensor_slices only includes the columns being used by DenseFeatures and no additional columns. Meaning the feature_columns list provided to the DenseFeatures init must include all the columns in the dataset, not a subset. Is this expected?

@tanzhenyu the original code to reproduce with DenseFeatures still fails but at a new location. Now the code fails at model.fit() with the following error:
2020-04-01 11:56:24.474889: W tensorflow/core/framework/op_kernel.cc:1706] OP_REQUIRES failed at cast_op.cc:124 : Unimplemented: Cast string to float is not supported
Traceback (most recent call last):
  File "/home/nikhil/workspace/testing_ground/test_tf_bug/check.py", line 86, in <module>
    model.fit(train_ds, epochs=1)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 71, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 920, in fit
    tmp_logs = train_function(iterator)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 608, in __call__
    result = self._call(*args, **kwds)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 672, in _call
    return self._stateless_fn(*args, **kwds)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/opt/anaconda3/envs/tf-nightly/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnimplementedError:  Cast string to float is not supported
	 [[node model/Cast_1 (defined at home/nikhil/workspace/testing_ground/test_tf_bug/check.py:86) ]] [Op:__inference_train_function_941]

Function call stack:
train_function

The code goes through if the tf.data.Dataset.from_tensor_slices only includes the columns being used by DenseFeatures and no additional columns. Meaning the feature_columns list provided to the DenseFeatures init must include all the columns in the dataset, not a subset. Is this expected?

Can you use print(tf.version) and let us know the output?

tf.__version__

tf.__version__ is the following : '2.2.0-dev20200331'

That's odd. The results I'm getting is:
tensorflow version 2.2.0-dev20200331
wrapper ||call - predict|| = 0.0
wrapper predict = [[90.]] ; correct is 37+511= 76
wrapper call = [[90.]] ; appears to do 5*(7+11)= 90

Ah I see, that's the code snippet provided by @MartinMohlenkamp, yup that one returns the result you provided. I was referring the original code snippet I provided in the issue description above. The snippet is larger, so my apologies, but that's the one that's failing.

@tanzhenyu note that when you ran my code snippet, call and predict agreed, but both gave the incorrect output.

Ok then it looks like there's been some changes regarding the unified model call path. Re-assigning to @qlzh727

@nikhil-mathew-td  Closing this issue as it is fixed in latest TF-nightly version.Thanks!

From above discussions I don't think this is fixed

@nikhil-mathew-td Looks like this was resolved in recent tf-nightly. I cannot reproduce the issue. Output from call and predict are same.
Here is the gist with the @nikhil-mathew-td code.
Here is the gist with @MartinMohlenkamp code.
Please verify once and close the issue if this was resolved for you. Thanks!

In the gist of my code, call and predict agreed, but both gave the logically incorrect output.
The effect is not a dot product 3*7+5*11 but is instead ignoring the 3 and doing 5*(7+11).
This is the same state as on April 1.
Or perhaps the line
xs = [base(layers.Lambda(lambda x : x[:,:,i],name='view'+str(i))(parallel_inputs)) for i in range(Nparallel)]
does not slice as I thought and a now-fixed bug in predict made me think it was correct.

Any progress on that issue? I am still getting inconsistent behaviour in tensorflow 2.3.1 when using batches.

In TF 2.5 I get the same result as my July 31, 2020 comment: call and predict agree, but both give what I thought was a logically incorrect answer. If my logic is correct, then the bug is in tensorflow.karas.layers.Lambda rather than call or predict.
I don't have a convenient way to test 2.6 nightly.

Hi There,
This is a stale issue. As you are using an older version of tensorflow, we are checking to see if you still need help on this issue. Please test the issue with the latest TensorFlow (TF2.7 and tf-nightly). If the issue still persists with the newer versions of TF, please feel free to open it in keras-team/keras repository by providing details about the issue and a standalone code to reproduce the issue. Thanks!
Please note that Keras development has moved to a separate Keras-team/keras repository to focus entirely on only Keras. Thanks!

I have the same issue even in the version 2.12.
Name: tensorflow-macos
Version: 2.12.0


Add Std Handle Values to WinSDK

Added the values of STD_*_HANDLE to WinSDK for use with GetStdHandle since otherwise swift complains about storing a negative number in a UInt32. See: https://docs.microsoft.com/en-us/windows/console/getstdhandle

Let's see if I have CI privileges yet:
@swift-ci please test

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge


Add footer before setting adapter to ListView

There was an issue where the Find Courses layout was not visible on devices containing OS version below 4.2. This has been handled.
Please review - @rohan-dhamal-clarice @hanningni
JIRA: https://openedx.atlassian.net/browse/MOB-1701

Nice. :+1:

:+1:


PDF document loader (JS) issue - readFile is not a function

I am trying to run the PDFLoader [example] using pdf-parse, and I encountered an issue in the browser:
Uncaught (in promise) TypeError: readFile is not a function
  at PDFLoader.load (langchain_document_l…js?v=2ceaa6ca:38:22)
  at async loadAndSplitDoc (ResultPage.jsx:35:17)

Going through others who shared similar issues, all i could get from it was that it could be related to fs module but i don't know. I'm still relatively new to Javascript and Langchain so this could be a dumb question (sorry in advance if it is)
Here's my code and my file structure for context:
const loadAndSplitDoc = async () => {
    const loader = new PDFLoader("doc.pdf");
    const doc = await loader.load();
    console.log(doc);

    const textSplitter = new RecursiveCharacterTextSplitter({
      chunkSize: 1000,
    });
    const docsSummary = await textSplitter.createDocuments([doc]);

    return docsSummary;
  };

@dosu-bot I am having the same issue, when i try the following code inside a "jest" test.
const loader = new PDFLoader("doc.pdf");
const doc = await loader.load();

Here's my jest config:
module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  collectCoverageFrom: [
    'src/**/*.{ts,tsx}',
    '!**/node_modules/**',
    '!**/vendor/**',
  ],
  moduleDirectories: ["node_modules", 'src'],
  moduleFileExtensions: ["ts", "tsx", "js", "jsx", "json", "cjs", "mjs"],
  verbose: true,
  moduleNameMapper: {
    "fs/promises": "<rootDir>/node_modules/fs-extra/lib/fs"
  },
  transform: {}
};

Here's how i start the tests : node --experimental-vm-modules node_modules/jest/bin/jest.js


babel compile fails if no .babelrc

This is new in 7.12.0.
PR coming.

released https://github.com/TypeStrong/atom-typescript/releases/tag/v7.12.1 :rose:


Add Sync bound for set_global_recorder

global recorder is shared between threads and must be sync
closes https://github.com/metrics-rs/metrics/issues/511

I spend some time thinking, if the Send bound is also needed. But seems not:
https://github.com/metrics-rs/metrics/blob/main/metrics/src/recorder/cell.rs#L44
global recorder is leaked via Box::leak and its destructor won't be called

Released as part of metrics<EMAIL_ADDRESS>Thanks again for your contribution!


Serialized objects should store the FTYPE used to create them

See issue #437 ... E.g., log binning created with FP32 and stored to disk doesn't load if running PISA in FP64 since to that precision, bin edges do not appear to be logarithmically spaced, but we'll want to be able to work with both.
There is the question of whether to convert the datatype to current PISA FTYPE upon loading an object stored using a different FTYPE. At least a warning should be provided, and the load should not fail as it does now (at least this seems reasonable to me).
While this arose with @philippeller fixing binning to actually use the chosen FTYPE, this same behavior should probably be propagated to all objects that can be serialized. (At the very least, store the FTYPE used and we can decide how to use that information later.)
The implications are pretty far-reaching, though; all objects that can be instantiated from serialized objects would need to add e.g. an optional ftype argument (that defaults to pisa.FTYPE). This doesn't seem too onerous, but it does change the interfaces for probably a bunch of objects. And then there's the issue of how to work with legacy objects that have already been serialized to disk (i.e. that don't have the ftype field in them). If simply loaded, which is done by loading the JSON and then instantiating the object essentially via **kwargs, then the ftype arg will come from the default, and could contradict the data type actually used to create the object. Any further assumptions that depend on ftype might then fail, unless we do some hacks. (E.g., the workaround for binning could be to check for log spacing in FP32 precision even if ftype is FP64.)

Sure, that should probably be the case, but it didn't cause any harm, so getting rid of it for now..


Enhance user authentication, performance, and documentation updates

Pull Request 195
This PR is automatically created for the new branch djh00t-patch-2 created by @djh00t
Summary
This pull request encompasses a series of enhancements and updates focused on user authentication, performance improvements, and documentation revisions. Key additions include the creation of essential documents like CONTRIBUTING.md, CODE_OF_CONDUCT.md, and LICENSE.md, while also addressing minor bugs and refining the user experience during authentication.
Furthermore, the codebase has undergone significant refactoring for increased readability and maintainability. Overall, these changes aim to bolster the application's functionality and ensure that contributing guidelines are clear and comprehensive for future developers.
Motivation and Context

Enhanced user authentication to improve security and user experience.
Refactored code for better readability and maintainability.
Improved performance across several modules to ensure efficiency.
Updated documentation to reflect recent changes and provide clearer guidance.
Created essential files like CONTRIBUTING.md, CODE_OF_CONDUCT.md, and LICENSE.md for project clarity and compliance.
Fixed bugs to ensure a more stable codebase and user experience.

Types of Changes

[ ] 📚 docs: Documentation changes

docs(CONTRIBUTING): Create CONTRIBUTING.md

[ ] ⚠ other: Changes that don't fit into the above categories

Merge branch 'release' into djh00t-patch-2
Chore: Enhance user authentication, performance, and update documentation (#193)
docs( CODE_OF_CONDUCT.md ): Create CODE_OF_CONDUCT.md
Chore: Enhance user authentication, improve performance, and update docs (#192)
doc(LICENSE.md): Create LICENSE.md
Improve documentation and fix bugs in user authentication system (#183)
Improve performance and refactor code for better readability (#181)
Release (#179)
Improve documentation and fix minor bugs in the codebase (#176)
Refactor styles and improve documentation for code consistency (#170)
Add new user authentication feature and improve error handling (#158)
Refactor code for improved readability and maintainability (#147)
Improve performance and fix bugs in user authentication module (#145)
Update build configuration and enhance documentation for features (#138)
Update authentication process and improve user experience (#135)
Update user authentication feature. (#133)
Update documentation and fix the build configuration (#130)
Update authentication flow and improve error handling (#120)

Closing PR - its a duplicate.


[Improvement][Dependent] The project list in dependent node's permissions should not be restricted.

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
The project list in dependent node's permissions should not be restricted. Suggesting create a new API without project permission restrictions and provide it to dependent nodes in UI.
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I want to accept it, please assign to me

I want to accept it, please assign to me

I've assigned to you. Looking forward to your contribution.


type hint with "Union" operator doesn't work properly

Environment data

Language Server version: 2022.11.20 (pyright 470188d0)
OS and version: windows 11
Python version (& distribution if applicable, e.g. Anaconda): 3.10, 3.11

Code Snippet
I made simple function which returns identity matrix like below
vector = list[float | int]
matrix = list[vector]

def mat_identity(n: int) -> matrix:
    """
    returns n by n sized identity matrix
    """

    I = [[1 if i == j else 0 for j in range(n)] for i in range(n)]
    return I

Repro Steps
just type the code
Expected behavior
obviously the function shouldn't have any problem, as code below returns no error at all
vector = list[float | int]
matrix = list[vector]

m: matrix = [[1, 0, 0], [0, 1, 0], [0, 0, 1]]

Actual behavior
pylance warn error like below
Expression of type "list[list[int]]" cannot be assigned to return type "matrix"
  "list[list[int]]" is incompatible with "matrix"
    TypeVar "_T@list" is invariant
      "list[int]" is incompatible with "vector"
        TypeVar "_T@list" is invariant
          Type "int" cannot be assigned to type "float | int"

Yes, this behavior is correct. To understand why, please review this documentation about static type concepts.


I could not build dlib in linux，how fix the issue:Cannot find source file: external/libpng/png.c

Building package dlib[core]:x64-linux...
-- Using cached /home/justin/Coding/vcpkg/downloads/davisking-dlib-v19.16.tar.gz
-- Extracting source /home/justin/Coding/vcpkg/downloads/davisking-dlib-v19.16.tar.gz
-- Applying patch fix-mac-jpeg.patch
-- Applying patch fix-sqlite3-fftw-linkage.patch
-- Using source at /home/justin/Coding/vcpkg/buildtrees/dlib/src/v19.16-9e1923f4db
-- Configuring x64-linux-dbg
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:56 (message):
Command failed: /home/justin/Coding/vcpkg/downloads/tools/cmake-3.14.0-linux/cmake-3.14.0-Linux-x86_64/bin/cmake;/home/justin/Coding/vcpkg/buildtrees/dlib/src/v19.16-9e1923f4db;-DDLIB_LINK_WITH_SQLITE3=ON;-DDLIB_USE_FFTW=ON;-DDLIB_PNG_SUPPORT=ON;-DDLIB_JPEG_SUPPORT=ON;-DDLIB_USE_BLAS=ON;-DDLIB_USE_LAPACK=ON;-DDLIB_USE_CUDA=OFF;-DDLIB_GIF_SUPPORT=OFF;-DDLIB_USE_MKL_FFT=OFF;-DCMAKE_DEBUG_POSTFIX=d;-DCMAKE_MAKE_PROGRAM=/home/justin/Coding/vcpkg/downloads/tools/ninja-1.8.2-linux/ninja;-DCMAKE_SYSTEM_NAME=Linux;-DBUILD_SHARED_LIBS=OFF;-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=/home/justin/Coding/vcpkg/scripts/toolchains/linux.cmake;-DVCPKG_TARGET_TRIPLET=x64-linux;-DVCPKG_PLATFORM_TOOLSET=external;-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON;-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON;-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON;-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE;-DCMAKE_VERBOSE_MAKEFILE=ON;-DVCPKG_APPLOCAL_DEPS=OFF;-DCMAKE_TOOLCHAIN_FILE=/home/justin/Coding/vcpkg/scripts/buildsystems/vcpkg.cmake;-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON;-DVCPKG_CXX_FLAGS=;-DVCPKG_CXX_FLAGS_RELEASE=;-DVCPKG_CXX_FLAGS_DEBUG=;-DVCPKG_C_FLAGS=;-DVCPKG_C_FLAGS_RELEASE=;-DVCPKG_C_FLAGS_DEBUG=;-DVCPKG_CRT_LINKAGE=dynamic;-DVCPKG_LINKER_FLAGS=;-DCMAKE_INSTALL_LIBDIR:STRING=lib;-DCMAKE_INSTALL_BINDIR:STRING=bin;-DDLIB_ENABLE_ASSERTS=ON;-G;Ninja;-DCMAKE_BUILD_TYPE=Debug;-DCMAKE_INSTALL_PREFIX=/home/justin/Coding/vcpkg/packages/dlib_x64-linux/debug
Working Directory: /home/justin/Coding/vcpkg/buildtrees/dlib/x64-linux-dbg
Error code: 1
See logs for more information:
/home/justin/Coding/vcpkg/buildtrees/dlib/config-x64-linux-dbg-out.log
/home/justin/Coding/vcpkg/buildtrees/dlib/config-x64-linux-dbg-err.log
Call Stack (most recent call first):
scripts/cmake/vcpkg_configure_cmake.cmake:282 (vcpkg_execute_required_process)
ports/dlib/portfile.cmake:33 (vcpkg_configure_cmake)
scripts/ports.cmake:71 (include)
Error: Building package dlib:x64-linux failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: dlib:x64-linux
Vcpkg version: 2018.11.23-unknownhash
Additionally, attach any relevant sections from the log files above.
how could I give vcpkg the png.c file Manual？？
config-x64-linux-dbg-err.log
config-x64-linux-dbg-out.log

Hi @zdptrsz, thanks for reporting this issue!
I checked the port log in git and made sure it was fixed in #6292.
The only problem is that the COPYRIGHT is uppercase in portfile.cmake.
After my PR merge, please update vcpkg and rebuild dlib.
Thanks.

PR has been merged. Close this issue.


LAB_04

English:
Note: Wait for the Remote Desktop session and Server Manager to load.
Current Translation:
Remarque : attendez que la session Bureau à distance et que le Gestionnaire de serveur se chargent.
Suggested Translation:
Remarque : attendez que la session Bureau à distance et le Gestionnaire de serveur se chargent.
Description:
Change made for stylistic reasons

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

Run the bugfix tool and waiting for result. ICMS QUERY

Manually checked and modified


Problems with running docker image on windows 10

Edition	Windows 10 Home Single Language
Version	21H1
OS build	19043.1586
I was following instructions from the docs to install and run scancode with docker. Up to building the image, there was no issue, but when I am trying to run the image using docker-compose up  scancodeio-worker-1 exits with code 127 and nginx exits with code 1. I have docker desktop installed, and I have installed GNU make with chocolatey in my machine.
Here is the terminal message:
Container scancodeio-redis-1  Created
Container scancodeio-db-1  Created
Container scancodeio-web-1  Created
Container scancodeio-worker-1  Created
Container scancodeio-nginx-1  Created
Attaching to scancodeio-db-1, scancodeio-nginx-1, scancodeio-redis-1, scancodeio-web-1, scancodeio-worker-1
scancodeio-db-1      |
scancodeio-db-1      | PostgreSQL Database directory appears to contain a database; Skipping initialization
scancodeio-db-1      |
scancodeio-redis-1   | 1:C 22 Mar 2022 15:28:16.650 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
scancodeio-redis-1   | 1:C 22 Mar 2022 15:28:16.650 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
scancodeio-redis-1   | 1:C 22 Mar 2022 15:28:16.650 # Configuration loaded
scancodeio-redis-1   | 1:M 22 Mar 2022 15:28:16.651 * monotonic clock: POSIX clock_gettime
scancodeio-redis-1   | 1:M 22 Mar 2022 15:28:16.652 * Running mode=standalone, port=6379.
scancodeio-redis-1   | 1:M 22 Mar 2022 15:28:16.652 # Server initialized
scancodeio-redis-1   | 1:M 22 Mar 2022 15:28:16.652 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
scancodeio-redis-1   | 1:M 22 Mar 2022 15:28:16.653 * Ready to accept connections
scancodeio-db-1      | 2022-03-22 15:28:16.903 UTC [1] LOG:  starting PostgreSQL 13.6 (Debian 13.6-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
scancodeio-db-1      | 2022-03-22 15:28:16.904 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
scancodeio-db-1      | 2022-03-22 15:28:16.904 UTC [1] LOG:  listening on IPv6 address "::", port 5432
scancodeio-db-1      | 2022-03-22 15:28:17.092 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
scancodeio-db-1      | 2022-03-22 15:28:17.272 UTC [27] LOG:  database system was shut down at 2022-03-22 15:28:03 UTC
scancodeio-db-1      | 2022-03-22 15:28:17.355 UTC [1] LOG:  database system is ready to accept connections
scancodeio-web-1     | /usr/bin/env: ‘python\r’: No such file or directory
scancodeio-web-1 exited with code 127
scancodeio-worker-1  | /usr/bin/env: ‘python\r’: No such file or directory
scancodeio-nginx-1   | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
scancodeio-nginx-1   | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
scancodeio-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
scancodeio-nginx-1   | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
scancodeio-nginx-1   | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
scancodeio-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
scancodeio-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
scancodeio-nginx-1   | /docker-entrypoint.sh: Configuration complete; ready for start up
scancodeio-worker-1 exited with code 127
scancodeio-nginx-1   | 2022/03/22 15:28:25 [emerg] 1#1: host not found in upstream "web:8000" in /etc/nginx/conf.d/default.conf:2
scancodeio-nginx-1   | nginx: [emerg] host not found in upstream "web:8000" in /etc/nginx/conf.d/default.conf:2
scancodeio-nginx-1 exited with code 1

@pombredanne I am also facing the same issue. Can you please look into this?

@PlabanKr I was also facing the same issue. I fixed it by going to the manage.py file and at the end of the shebang line, just put a space at the end of the python word. This is an error of CRLF/LF.

@PlabanKr @Pratyush1606 any changes we can make in the codebase to fix this issue for good?

A warning about the crlf issue was added in the documentation https://github.com/nexB/scancode.io/commit/37d325a03c02dc5ae9205d4b16eaf729253523ee#diff-d9b149498982c0663c3b7170398773361ed5678f1a627e9c2fd8d2c955c563dbR38-R41