WIP: Add Azure Developer Credential

add azure developer credential and implementation
add unit test
change readme
WIP: Testing
Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Identity

I am concerned with credentials that spawn processes. I think it would be good to have an architecture meeting discussion about this design.

I would love to discuss in person why we need to be spawning processes.

Azure CLI, PowerShell, and Visual Studio credentials also do this.

Azure CLI, PowerShell, and Visual Studio credentials also do this.

These are apps we don't own. For azd, why can't we have a library that we can call in-proc?

These are apps we don't own. For azd, why can't we have a library that we can call in-proc?

Azure Developer CLI is written in Go. While it seems possible to compile this down to a native .dll or .so file there would be drawbacks. First we would need to package the native dependency, rather than relying on the executable, which already must have been installed, being in PATH. Also, we would need to limit the library to c-types, and deal with the native interop in all 4 languages this is being implemented in. The developer experience would be degraded in the same way it is for any native dependency. Given that we already use subprocess execution for all the other development time credentials, I think it makes the most sense here.

I had an offline discussion with @schaabs. We agreed to ship a preview as-is and then dig into pros and cons of the alternative approaches later.


Grib processed data

handle type of processed data for GRIB export

@marqh Minor tweak and we're there :+1:


Denoising

Your project is fantastic, quite a achievement! I saw another project which is similar to yours, they talk about denoising here in this issue:
https://github.com/hoverinc/ray-tracing-renderer/issues/2
I thought you might have experience in denoising techniques and want to add to the discussion. It would be cool to also see denoising ending up in your project.

@kristiansvenssonwsp
Hello, thank you!
That's a cool project - I was unaware of it, thanks for the link.  Truth be told, denoising is the area where I have the least experience and knowledge.  Over the last 4 years I have tried to educate myself about basic to advanced general ray tracing - which, as you know, covers a lot of fields: light transport, material brdfs, path tracing, monte carlo integration (and probability/statistics), ray intersection, geometry, BVH acceleration, gpu parallelism, etc, the list goes on.  One of the subjects I haven't got around to yet is the art of denoising rendered images in real time.  I'm not even sure how it's done to be honest, but if I had to take an educated guess, I would guess that before you write a final color to each pixel in the fragment shader, you would take a peek at the pixel's neighbors above, below, left and right, then somehow average the results, kind of like a bi-linear filter.  But that is probably a gross over-simplification.  I wonder how are fine edges handled?  How are different materials handled if they are only specular and do not require any denoising vs. diffuse surfaces that require it?
I noticed in the discussion link you provided that you included some shadertoy simple denoising examples.  Thanks for those, that will help me get started on understanding what goes into a denoiser.  Hopefully I can learn from the actual shader source code and try some of it out on my own project.  I'll definitely let you know if I get something working.
At NVidia, they are not only using denoisers, but somehow using machine learning, or AI, to assist with cleaning up the diffuse surface noise in real time.  There's been a lot of research in image recognition AI and it's only getting better each month it seems.  Exciting times!

"AI" sure is interesting.  Thanks for those links.  Since many path tracers are progressive just for the fun of watching, I was thinking why not just denoise every n frames, progressively increasing the denoising frame gap.  ZUH!
I wrote a basic PT (in BASIC, no less) .. only spheres.  I have a very simplistic approach to problem solving in many of my ideas.  Can't remember how I handled fireflies, but my russian roulette technique was unique.
Of course this project's caustics rendering and "firefly handling," if that's what one might call it, is on a completely different level.  My favorite scenes are the geometry showcase and difficult lighting.
I have one thing to contribute, and maybe more as time goes on.  In the programming forum I visit, there has been discussion of PRNGs.  I invented a couple.  My new one seems to favor monte carlo path traching.
a *= a
a xor= w
w += 1

@dafhi
Hi David, yes I think that the state-of-the-art denoisers actually do their work over multiple frames like you envisioned with the progressive rendering.  The big difference with what they do and simple progressive rendering like I do is: I just present the 1 sample per pixel to the screen every frame from startup, that's why you see the initial noise when you start moving the camera.  When the camera is still, the 1 sample per pixel is kept and then averaged with the new 1 sample per pixel, which becomes essentially '2 samples' per pixel, and so on and so on...  I think the AI denoisers and other sophisticated temporal denoisers actually take the first 1 sample per pixel image as input but it's not presented to the screen right away.  Some magic happens under the hood  with the following 1 sample per pixel image and when it's done, the new combined, denoised image is actually presented to the screen.  Of course all of this happens at a blazing 60 times a second.  Not sure yet how it al works, I'm still looking into those links that Kristian provided above.
That's awesome that you wrote a path tracer in BASIC!  I didn't think that something like that was even possible!  Takes me back to my first programming experience, programming in BASIC on my beloved Commodore 64 back in 1983!  Yes, the caustics and fireflies have been a 'thorn in my side' ever since this project started.    The reason why those happen in a pure basic path tracer without direct lighting sampling is that a secondary GI diffuse surface has found a bright light by chance.  Say your camera ray hits a diffuse surface, then it bounces from that to another diffuse surface, and then to another diffuse surface, and then from there it accidentally hits a light source.  Well if you work your way back, that contributes a really bright pixel for that little point on the initial diffuse surface that the camera ray hit.  And its neighbors are not so lucky, maybe they hit another secondary diffuse surface as well, but odds are that the ray won't find the light source after all that random diffuse bouncing, so it just keeps bouncing and attenuating the ray's contribution until it either dies out or escapes the scene, resulting in a darker pixel right around the super bright one that was 'lucky' (unlucky for us! ha).  The only way to get around this is to implement direct light sampling, then handle fireflies by saying since I've already manually added the light contribution on every diffuse bounce, then if a diffuse surface accidentally hits a light source, the ray is terminated and the contribution is 0.  This keeps everything much more uniform and you can control the amount of brightness and shadows in the scene much easier (and have faster convergence too!).  The good news is that it has been mathematically proven even though if you 'cheat' and add direct light sampling, that your Monte Carlo path tracer is still consistent and totally unbiased.  It's one of those amazing mathematical quirks with statistics and probability theory.
Thanks for the PRNG tidbits, I must admit that the subject of generting random numbers is way above my head.  I have never really looked deeply into how that all works.  I credited iq on ShaderToy for his amazing bit-shifting generator that runs really fast inside a shader on the GPU.  I owe a lot to those 4 little magic lines of code (in terms of smooth randomness for the Monte Carlo bits of my project, which is so important), but I have no idea how they work! ;-D

@erichlof perhaps this blog can shed some more light on the subject. See, most denoising algo's are actually opensourced, for instance nvidia's SVGF implementation is opensourced in the Quake II repo.
Also, this video was mighty intersting to watch

@MaartenBreeedveld
Thank you for the links!  I'm going to read the blog post right now.  Also I have seen that video briefly before (I'm always looking for newly uploaded PathTracing videos on YouTube, ha ha).  But I will take a closer look at it with the blog post fresh in my memory ;-)
Thanks again!
-Erich

@erichlof
I stumbled on an A-SVGF example on ShaderToy I can't say it's fully working, but it's looking promising!

@MaartenBreeedveld
Ah yes I had seen this one before and had actually bookmarked it.  Although it is impressive in quality, it is equally impressive in how dense the code is.  His coding style, cryptic variable naming, and lack of white space is a huge barrier for me understanding what is going on.  Obviously he is very knowledgeable (you could even say brilliant),  but without meaningful code comments or more thoughtful structure, I would be at a loss if I just started poking around at variables and magic numbers.  I wouldn't know if anything I was doing was changing or breaking anything.
That being said, I am going to go back and take a closer look at the original implementation - the one used for Quake II RTX.  Here's an NVIDIA link.  Also, here's an interesting link that uses much simpler scenes and geometry:
CUDA Github project where I might be able to better separate the denoising parts from the usual rendering parts.
As always, thank you for the links and heads-up about these resources!

@erichlof
I must agree with you there, I'm pretty much a noob when it comes to shaders, but I've earned some stripes in programming.
The math on this shader looks very dense indeed. I was somewhat hoping it was less of an issue for you considering your knowledge about shaders.
The NVIDIA link looks good indeed.
Thank you very much on the explanation of the channels, super interesting!

@MaartenBreeedveld
No problem!  I'm glad to try and explain - it forces me to make sure I understand it myself first! LoL
Yes it is unfortunate that I can't wade through his Shadertoy code as it is probably exactly what my renderer needs right now.  I'm the type of person who goes back and reads introductory chapters about vector operations (like dot and cross products) because I want to make sure I understand why it is done a certain way, and often times having it explained by 2 or 3 different authors with different approaches/diagrams helps me to view it from a new perspective and hopefully solidifies my understanding of the topic further.
I'll openly admit to copying and pasting from StackOverflow and Shadertoy at times.  But I never just drop it in and leave it - I play around with it, poking and prodding it (like a kid engineer with a toy car), most often breaking it, then trying to put it back together in a new way.  Then I can confidently make it a part of my codebase because I know that if something goes wrong with it, I'll at least have somewhat of an understanding of how and why it works.  For instance, it took me months to understand what a BVH is and how to create one and then how to traverse one in a shader.  I started by copying a couple of C++ BVH builders out there in the wild, then going line by line, breaking things and poking around until I could build my own.  It will most likely be the same for a denoiser. At this point I'm just trying to find a simpler one that I can play around with, maybe not even a full denoiser, but maybe just a basic spatial blur filter that I can drop into a shader and then improve upon, bit by bit.
You mentioned that you were newer to shaders.  I think you'll like this YouTube channel  .  If you check out the older ones, he explains in a very clear way how shaders work and how to do simple effects and basic shapes with ray marching.  His videos progress all the way to some pretty advanced concepts.  But overall, his videos are understandable and really well done.

@erichlof
Awesome! I've stumbled into that youtube channel in the past, it's a great resource indeed!
I'm going to have to free up some time to properly learn shaders I guess!
Anyway, I will be following your progress closely 👍!

@erichlof
https://www.shadertoy.com/view/3dd3Wr
https://github.com/BrutPitt/glslSmartDeNoise
Have a try.

Hi Erich,
I popped by to see how your project is going and I must say I'm super impressed with the denoiser!
Also, those NVIDIA guys didn't do it in one day. They were researching the matter for years as well :).
Awesome!

This YouTube video shows a comparison of the results of several denoising methods (SVGF, NFOR, ONND, MR-KP, BMFR, NBG):
https://youtu.be/9PVR1-GTt6g?si=crasv6MpmPudwiJV

@giovanni-a Thank you for the links and video!  That online viewer is a great tool - haven't seen many research papers use such an interactive tool for comparisons.  It really helps to show the differences.  Thanks again!

Thank you, @erichlof! Your work is fantastic, and I cannot wait to see what you will achieve next. I feel that the day when we’ll have a path tracer that works in real-time, with minimal noise, and even on mobile, is getting closer. Keep up the great work!

I've been following your project on and off for a few years now, and came here to make an issue about denoising myself and get a conversation going. So glad it's here! :D I'm sure you're very busy and it can be quite an undertaking, but I think implementing some sort of TAA reprojection and denoiser would bring this project to another, another level. With such good frame rates even on mobile, I'd love to try and use your renderer in the wild on something but (almost) everything has pesky motion!
I might take a stab at it myself if I get started on something. This looks fairly* easy* to implement (maybe).
https://www.shadertoy.com/view/WdjcDd
Thanks for your awesome work @erichlof !

@jerzakm Thank you very much for the kind words and suggestion! And thank you for the great example - that will be very helpful!
Sorry for the late replies - for the last couple of months, I have been going down the rabbit hole of trying to efficiently raycast a torus (which is traditionally very difficult and expensive because it is a surface of degree 4, or quartic). Sometimes I feel like the comical mad scientist who is creating solutions, throwing them out in disgust, starting over, try something else, throw that code out, and finally... - I have found a solution! I will be releasing a demo soon that ray traces and renders a very close torus approximation, for the cost of ray tracing a sphere! Therefore, we can have as many torus shapes as we want in a scene, and it'll run smooth on every device - even on mobile!
About the TAA denoising, I'm so glad you entered the discussion, @jerzakm , because I've tried looking at the freely available code for the complete A-SVGF denoising solution a couple of times in the past, but I just couldn't wrap my head around it - even when there was a couple shadertoys that demonstrated this method. And as I previously mentioned in this thread, before I just drop someone else's solution into my project, I feel the need to really understand it, line by line. That way, if I add something else in the future that conflicts/crashes with the denoiser or causes artifacts, I can debug it and at least know where to start in order to find a solution. And in the past, since I couldn't wrap my head around how A-SVGF does its magic,  I couldn't bring myself to just dump it in my codebase and start trying to hook it up to everything else.
However, your helpful shadertoy example that you linked to is very tight and focused, and therefore the amount of shader code is much more manageable for me to pour over. I'm glad you suggested the exact technique by name, TAA, because that is what the example focuses on for the denoising. If I'm not mistaken, A-SVGF uses a very similar component - TAA, but it is one of several components (in addition to the edge detection component, the screen-spatial filtering component, etc.) and is wrapped up and entangled in the A-SVGF source code, making it more difficult for me to tell what is what.
But now I feel I can study this small component (TAA with re-projection) of the overall larger denoising solution, in order to understand it more fully. And hopefully soon, I can start experimenting (back to my 'laboratory' 😆) with hooking up TAA to my own renderer and customizing it.
But don't let this stop you from trying it on your own! If you do, and you are able to get it working, please share your findings on this thread here, and maybe even share a GitHub Gist, in order to see how you hooked it up. Like I mentioned, I'm working on the torus raycasting at the moment,  but soon when that's finished (or good enough to be satisfied,  ha), then I will go down the TAA rabbit hole! Several people have asked for a more sophisticated denoising scheme, and I really appreciate the look of the A-SVGF approach - It seems to be the standard that most ray tracers turn to (if one is not using neural networks and AI to denoise the final image). So I think that if I could get a small but essential component of that solution working, it would benefit our renderer a lot! 😊
-Erich


Rollup needs to be installed globally

The getting started instructions don't mention that rollup needs to be installed globally. magic-script build -i after magic-script init fails otherwise.
Alternately, you could change the init'd build script to run npx rollup -c instead of just rollup -c.

Thanks @brianpeiris. I'll look at this.

It shouldn't fail if you run npm install first since we include rollup ad dev dependency and run it usingnpm run`


Update aws-java-sdk-dynamodb to 1.12.181

Updates com.amazonaws:aws-java-sdk-dynamodb from 1.11.1034 to 1.12.181.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.amazonaws", artifactId = "aws-java-sdk-dynamodb" } ]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #263.


Ignore HTTP-date values with invalid year field

Some clients request PPD files with a random value (e.g.., 25 Nov 4461684 22:08:41) in the If-Modified-Since header (see Issue #5511).
Ignore this by rejecting invalid (RFC 7231 says the year field is 4DIGIT) years in httpGetDateTime().

Fixed in OpenPrinting CUPS.


Ignore CARGO_TARGET_DIR if it's the empty string

This change prevents CARGO_TARGET_DIR= cargo clean from wiping out the entire
working directory without warning.
Fixes issue #7551.
I'm not sure what's the intended behavior when CARGO_TARGET_DIR is ., so I left it alone:
maia@fluff (/tmp/tmp.9M7LYP7wJp)$ CARGO_TARGET_DIR=. cargo clean
error: could not remove build directory

Caused by:
  failed to remove directory `/tmp/tmp.9M7LYP7wJp/.`

Caused by:
  Invalid argument (os error 22)
maia@fluff (/tmp/tmp.9M7LYP7wJp)$

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Eh2406 (or someone else) soon.
If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.
Please see the contribution instructions for more information.

Thanks for the PR! This feels like a sort of odd inconsistency though in Cargo, since Cargo generally doesn't ignore unset environment variables anywhere else.
I would personally prefer that we not do this since there's a whole number of ways Cargo could cause problems given various env vars, but we can't really protect against all of them.

I don't have strong feelings about this particular solution. But it does feel wrong to me that an empty string is considered a relative path to the working directory, especially with a dangerous command like cargo clean. (Compare EMPTY= rm -r "$EMPTY", for example.)

If this wants special treatment for specifically just cargo clean then I think the treatment should happen there, otherwise this is affecting a pretty core part of Cargo about how the target directory is inferred, which strays away from the original goal about helping with cargo clean.


UNAVAILABLE

Operating System version: ubuntu 18.04
Firebase SDK version: firebase-admin 7.0.1
Firebase Product: FireStore

I had this issue a long time ago and I was told it's an internal issue that should be fixed. It was fixed as I didn't see it for a long time but I got it twice already in the last week.

java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564)	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:545)	at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:86)	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)	at com.app.API3.addTransaction(API3.java:3474)

Relevant Code:
It happens with the following code
future = batch.commit();
future.get();    <<<<< failing code

Thanks

@Amosss We currently retry Commits that fail with UNAVAILBLE (https://github.com/googleapis/nodejs-firestore/blob/master/dev/src/v1/firestore_client_config.json#L67).
These retries are scheduled with backoff, but if the retries don't succeed after 10 minutes, the operation is cancelled and the Promise that is returned is rejected. Do you notice these 10 minute timeouts in your setup? If not, there might be an issue with our backoff logic that we should look at.

Thank you for your reply.
From what I gathered, a user tried to do a certain action and the response was this exception. Obviously it doesn't happen most of the time, it's just it did happen twice last week so I thought to bring it up here.
Bottom line, I don't know if the 10 minutes timeouts occurred or not.

Thanks for the update. Unfortunately, there won't be much we can do right now. If this issue gets reported again, it would help if you could give us information regarding the timeout (and debug logs if available).


Params registered on app available in routers used by app

I discovered today that it seems params registered on app are not respected in routes registered via a router (express.Router()).  Perhaps I'm just missing something though!
While sectioning an application's routes via routers is super handy, having to re-register shared params on each of them is not.
A use case to consider: an application that has a page which renders user details in which the view needs the user data, and in addition the app exposes an API to get the raw JSON response for use by a client-side app, mobile app, or third party.
var express = require('express');
var app = express();

// Shared Params

app.param('user', function (req, res, next, id) {
  req.user = { id: id, name: 'Fred' };
  next();
});

// Website

app.get('/users/:user', function (req, res) {
  res.render('user', req.user);
});

// Application API (JSON Responses)

var APIRouter = express.Router();

router.get('/users/:user', function (req, res) {
  res.json(req.user); // req.user is undefined!
});

app.use('/api', APIRouter);

// App Start

app.listen(8000);
````````

Seems that shared params across application "sections" when using routers would be a very common scenario.

---------

Maybe this is going too far, but it'd make sense that any param registered on a parent router applied to its children, but possibly not visa-versa:

``````javascript
var parent = express.Router();
var child = express.Router();

parent.param('foo', function (req, res, next) {
    req.foo = { ... };
    next();
});

child.param('bar', function (req, res, next) {
    req.bar = { ... };
    next();
});

parent.get('/:foo/:bar', function (req, res) {
    res.json({
        foo: req.foo, // OK!
        bar: req.bar // undefined
    });
});

child.get('/:foo/:bar', function (req, res) {
    res.json({
        foo: req.foo, // OK!
        bar: req.bar // OK!
    });
});

parent.use(child);

Thoughts/advice appreciated!

Anyone reaching this page after 2014, all you need to do is express.Router({mergeParams: true});

I love all of you...  express.Router({mergeParams: true}) did the trick!

Wait how does one install Express 4.5? I need this mergeParams feature :(

Yeah, i need it too.... im using multiple api point where i want to reuse all params like ":contentId", ":userID", ":albumId" or what eva, but also have automatic routing adding on top of app.... really need this :(

In my case even only using router itself is not even working:
`
router.param('id', function (req, res, next, id) {
Lesson.getLessonById(id, function (err, lesson) {
if (err) { throw err; }
req.lesson = lesson;
});
console.log(req.lesson); //undefined
next();
});
router.get('/dashboard/:id', function(req, res) {
        var lesson = req.lesson; //lesson undefined
        res.render('lessons/dashboard', { lesson: lesson });
});`

@iyunusov You are calling next() in the wrong place. It should be called after your req.lesson = lesson; line (INSIDE the async callback, not outside).

@iyunusov:
router.param('id', function (req, res, next, id) {
    Lesson.getLessonById(id, function (err, lesson) {
        if (err) { throw err; }
        req.lesson = lesson;
        console.log(req.lesson);
        next();
    });
});

router.get('/dashboard/:id', function(req, res) {
        var lesson = req.lesson;
        res.render('lessons/dashboard', { lesson: lesson });
});


PythonUE not Working as Intended with Packaged Project

I have an actor which gets its variables updated by PythonUE and then I use actor.post_edit_change_property('MyProperty', 0) which refreshes the actor and updates it in the world with the new properties. Everything works perfectly when I launch the game in the editor (as a standalone game).
The problem is that when I package the game, everything works except the refreshing of the actor. The actor variables get updated but the actor does not refresh. Note that I tried using actor.post_edit_change() and actor.pre_edit_change() and they do not refresh the actor even using the game from the editor.
A reference to another issue I opened on this last subject: #349
Any ideas? Thanks! :)

Hi is this a c++ actor ? if yes, can you post its declaration ?


Revocation in IdentityServer not revoking all instances of a subject

IdentityServer4 Issue
I have a case, through my use of Extension Grants, where I can mint multiple tokens with the same "subject" using different clientIds;

ClientId
Subject

client_one
subject_one

client_one
subject_one

client_two
subject_one

client_two
subject_one

When a revocation happens, the expected behavior is that all refresh_tokens that reference the subject must be revoked.  At the moment IdentityServer4's IStores only account for a single revocation, but we need full revocation across all clients.
I have an implementation that delegates everything to your dentityServer4.Contrib.RedisStore.Stores.PersistedGrantStore, except for;
        public async Task RemoveAllAsync(string subjectId, string _, string type)
        {
            var clientIds = await _clientStoreExtra.GetAllClientIdsAsync();
            foreach (var clientId in clientIds)
            {
                _persistedGrantStoreImplementation.RemoveAllAsync(subjectId, clientId, type);
            }
        }

You may want to account for this in future releases.
Here is my PersistedGrantStoreExtra implementation.

I will see how it goes for Identity server 4 contract, if it's going to introduce new method there to cater this case. Later I will add implementation for it maybe with additional Redis Set.
Thanks for your input !

This is implemented already with Version 4 of the library, I will close it


implement cast to decimal

continuation of PR #527 .   I tried to do a git rebase, and things got messy so I started over, and github closed the old PR, so I just created a new one.

Codecov Report

Merging #529 (78f5512) into main (1213d82) will decrease coverage by 0.00%.
The diff coverage is 92.00%.

@@            Coverage Diff             @@
##             main     #529      +/-   ##
==========================================
- Coverage   92.98%   92.98%   -0.01%
==========================================
  Files         175      175
  Lines       11182    11232      +50
==========================================
+ Hits        10398    10444      +46
- Misses        784      788       +4

Impacted Files
Coverage Δ

core/src/data/value/into.rs
96.55% <80.00%> (-0.82%)
:arrow_down:

core/src/data/value/literal.rs
94.68% <87.50%> (-0.21%)
:arrow_down:

core/src/data/value/mod.rs
95.29% <100.00%> (+<0.01%)
:arrow_up:

test-suite/src/function/cast.rs
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1213d82...78f5512. Read the comment docs.


Pass Array values through --set

Please add the ability to pass Array values through --set.
For example, the stable/aws-cluster-autoscaler Chart must receive the following:
autoscalingGroups:
  - name: my-asg-name
    maxSize: 50
    minSize: 2

If this is already possible, please do tell.. And update the docs to show how.

Array values can be specified using curly braces: --set foo={a,b,c}. This will translate to the YAML:
foo:
  - a
  - b
  - c

Maps have to be passed like this: something.key=value,something.key2=value2, which will translate to the YAML:
something:
  key: value
  key2: value2

The full documentation for --set is buried in this document: https://github.com/kubernetes/helm/blob/master/docs/using_helm.md#the-format-and-limitations-of---set

@technosophos Many thanks for responding.
After reading the referenced doc, I still don't see how to reference the following via --set:
autoscalingGroups:
  - name: my-asg-name
    maxSize: 50
    minSize: 2

Is  there a way to do this?

I believe that the most straightforward way to do that would be to alter the structure of values.yaml into something like this:
autoscalingGroups:
  "my-asg-name":
    maxSize: 50
    minSize: 2
  "my-other-asg":
    maxSize: 50
    minSize: 2

That strategy will simply make it easier to reference them from --set, while also being pretty easy to work with in templates.
As you probably noticed from the documentation, there is no way in the current --set syntax to access an array value by index. So working with arrays is sort of a pain.

So working with arrays is sort of a pain.

I'm looking for ways to set an array using set, but it's unclear to me what the above statement means? Is it possible to set an array with set, or isn't it? If it's sort of a pain, then I don't mind, but I'd like to find out how to do it.
I know I can convert the array to a map, but that's not really what I'm looking for, as the user experience (for an internal company-wide chart) degrades to work around a technical issue.
I'm not saying this should be possible, but if it's not, we should probably mention this clearly in the documentation. (and if it already is, then I'm sorry, I must have missed it)

From the documentation:

Lists can be expressed by enclosing values in { and }. For example, --set name={a, b, c} translates to:

name:
  - a
  - b
  - c

@pluttrell : Maybe
--set autoscalingGroups[0].name=my-asg-name --set autoscalingGroups[0].maxSize=50 --set autoscalingGroups[0].minSize=2

Hello all,
Do you know how about this structure of Kubernetes kind 'affinity'.
###################################################
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb-replicaset
topologyKey: kubernetes.io/hostname
##################################################
Should I write in
--set=affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution.labelSelector.matchExpressions.key=app,
affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution.labelSelector.matchExpressions.operator=In,
affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution.labelSelector.matchExpressions.values=mongodb-replicaset,
affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution.topologyKey=kubernetes.io/hostname
Or some other correct syntax?
Thanks very much.
Hong

I think that when you have to apply complexe modifications like these, you'd bettre have to put them in a file intead of using --set.

Hello @cabrinoob
Thanks for your recommendation.
That's true that I used --values=values.yaml to adapt appropriated value.
So there is no syntax to express affinity syntax by --set, right?
Thanks very much.
Hong

You do not have to have the same structure in your values.yaml file than in your deployment.yaml resource.
You can have a relatively "flat" structure in values.yaml that makes usage of --set more simple.

Hello @cabrinoob
Thanks for your comments.
But, I saw the most of charts application in https://github.com/kubernetes/charts/tree/master/stable
For example:
#############################################
values.yaml
affinity: {}
deployment.yaml
{{- with .Values.affinity }}
affinity:
{{ toYaml . | indent 8 }}
{{- end }}
############################################
Is that mean that deployment file needed to be patched when desired?
Thanks.
BR
Hong

Yes, that means the whole bloc named "affinity" will be taken like this
(toYaml / indent 8) and injected in the deployment.yaml.
But, I think that if the affinity bloc is "complex" you'd better have to
override it through a file instead of using --set because in that case
--set parameter can be very tricky.
Le 27 juin 2018 13:59, "Hong"<EMAIL_ADDRESS>a écrit :
Hello @cabrinoob https://github.com/cabrinoob
Thanks for your comments.
But, I saw the most of charts application in
https://github.com/kubernetes/charts/tree/master/stable
For example:
#############################################
values.yaml
affinity: {}
deployment.yaml
{{- with .Values.affinity }}
affinity:
{{ toYaml . | indent 8 }}
{{- end }}
############################################
Is that mean that deployment file needed to be patched when desired?
Thanks.
BR
Hong
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/helm/issues/1987#issuecomment-400646663,
or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE8VN2OAyYhurf0eA9zFLKHYjo39fOf4ks5uA3OmgaJpZM4MDqL3
.

Been gruelling over this all morning after trying to use the gcloud proxy: https://github.com/helm/charts/tree/master/stable/gcloud-sqlproxy
The first example that it gives is: [{instance: instance, project: project, region: region, port: 5432}] which is basically JSON. After lots of failed attempts and eventually figuring I need to escape the commas, I found that it's actually not the correct syntax after stumbling upon this issue.
 I was hoping to provide the whole instance array there but it looks like I'll have to hardcode it to a single instance using 4x ENV vars. Now I'm wondering why gcloud-proxy even bothered to make it an array cause it's impossible to provide dynamically

If someone googles how to pass imagePullSecrets via --set flag, here is the command:
For example, helm template with imagePullSecrets like this one:
spec:
    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
    {{- end }}

command:
--set 'imagePullSecrets[0].name=regcred'

@pkaramol I hop eyou already figured this out, but since the master.installPlugin value you're passing isn't quoted, it treats your argument to --set as master.installPlugins={kubernetes:1.21.2, , and then attempts to parse workflow-job:2.36'} as your next argument. You need to remove the space between list items as that space would be treated literally if the curly braces were properly quoted, but if you remove the space and are using a shell which would attempt to expand {1,2,3} (such as bash), you will also need to quote the outside of the curly braces (which also means you want to remove the single quotes from your array element values, unless the values should be strings which include literal quotes)

@intellix re Gitlab, I also stumbled across the need to pass an arbitrary number of environment variables to our helm chart whilst using set. I ended up with this hacky CSV to array solution but I'd be interested if you or anyone else had a better suggestion:
--set app.extraEnv="ENVVAR1=foo\,ENVVAR2=bar\,ENVVAR3=bat"
env:
{{- range (split "," .Values.app.extraEnv) }}
  - name: {{ (split "=" .)._0 }}
    value: {{ (split "=" .)._1 | quote }}
{{- end }}

变量
values.yaml示例
master:
    initContainerEnv:
      - name: http_proxy
        value: "http://<IP_ADDRESS>:3128"

变量配置方式
helm install jenkins-blueocean \
  --set master.initContainerEnv[0].name="JENKINS_UC" \
  --set master.initContainerEnv[0].value="https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json/" \
  --set master.initContainerEnv[1].name="JENKINS_UC_DOWNLOAD" \
  --set master.initContainerEnv[1].value="https://mirrors.tuna.tsinghua.edu.cn/jenkins/" \
  stable/jenkins

values.yaml示例
master:
    initContainerEnv:
      - name: http_proxy
        value: "http://<IP_ADDRESS>:3128"

变量配置方式
helm install jenkins-blueocean \
  --set master.initContainerEnv[0].name="JENKINS_UC" \
  --set master.initContainerEnv[0].value="https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json/" \
  --set master.initContainerEnv[1].name="JENKINS_UC_DOWNLOAD" \
  --set master.initContainerEnv[1].value="https://mirrors.tuna.tsinghua.edu.cn/jenkins/" \
  stable/jenkins

Thanks @willzhang . Your solution made by day <3

set using an array list destroy the values yaml structure

I have powershell script to set a collection of values to a single key in values.yaml

But when I execute the The script result like this.

Actually I want the values like this, But i got this structure only use the set flag like this helm upgrade testing testconfig --set "KeyvautCollection={may0,may1}"

I want to pass a dynamic array along with --set flag

@hari4147 stackoverflow.com is a way better site for this type of questions.

sorry for the inconvenience.
On Fri, 26 Aug 2022 at 8:29 PM, Davide Madrisan @.***>
wrote:

@hari4147 https://github.com/hari4147 stackoverflow.com is a way better
site for this type of questions.
—
Reply to this email directly, view it on GitHub
https://github.com/helm/helm/issues/1987#issuecomment-1228603283, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AI43BQVR77GDDCNHXDBXCJLV3DLV7ANCNFSM4DAOUL3Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

In case when a value could be either a single string, or multiple strings, using toStrings in a template makes it much easier to override.  In the following setup, it is possible to use --set foo.mylist=bar to override a,b,c list with a single value bar, while still treating it as a list.
values.yaml
foo:
  mylist: ["a", "b", "c"]

templates/foo.yaml
{{- range $val := .Values.foo.mylist | toStrings }}
  {{- $val }}
{{- end }}


merge from template-node-app

package.json

chore(deps): add ^11.10.3 remove ^11.10.2 (devDependencies.npm-pkgbuild)

:tada: This PR is included in version 6.4.4 :tada:
The release is available on:

npm package (@latest dist-tag)
v6.4.4
GitHub release

Your semantic-release bot :package::rocket:


Issue/16

          3 Messages

:book:
GitHub Issue: #issue/16

:book:
Executed 2 tests, with 0 failures (0 unexpected) in 0.519 (0.524) seconds

:book:
What a good PR :white_flower:

Develop PR Check Result

Item
Result

Head Branch check
:o:

Base Branch check
:o:

Merge Commits check
:o:

Modification Volumn check
:o:

  Generated by :no_entry_sign: Danger


Add valueLink to Input

https://facebook.github.io/react/docs/two-way-binding-helpers.html

valueLink is going to be removed from the react core. I wouldn't go into this direction.

Agree with @oliviertassinari Two way binding helpers sound like an antipattern in react from my POV.
Do you have a specific scenario where you are using them and they can't be removed? Removing valueLink from core means a lot

@oliviertassinari

ReactLink is just a thin wrapper and convention around the onChange/setState() pattern. It doesn't fundamentally change how data flows in your React application.

Where you read about removing valueLink from react core? proof?

@LestaD  https://github.com/facebook/react/blob/master/src/renderers/dom/client/wrappers/ReactDOMInput.js#L98

@oliviertassinari oh!


Useful generators for PLC and PIR to use in tests and get better coverage

[ ] https://github.com/IntersectMBO/plutus/issues/6126
[ ] https://github.com/IntersectMBO/plutus/issues/6127
[ ] https://github.com/IntersectMBO/plutus/issues/6128
[ ] https://github.com/IntersectMBO/plutus/issues/6129

I don't find the existence of this issue helpful, so I'm closing it. Feel free to reopen if you disagree. Also we can just use labels instead of opening such issues, so maybe we should discuss that.


keepassxc

Checklist

[X] I have checked the moderation queue that no new version has been submitted (only visible when logged in), or the submitted package version has failed the automated checks.

Please navigate to the following link to view the moderation queue.

[X] I have verified that the new version is a stable release.
[X] I have looked to see if there are any Bug reports that prevents a new version being submitted and that no existing Outdated Reports have been created.
[X] I have verified the new version has a Windows binary (EXE or MSI installer, Zip or other archive, etc).

New Software Version
2.7.0
Download location
https://github.com/keepassxreboot/keepassxc/releases/download/2.7.0/KeePassXC-2.7.0-Win64.msi
Package Page
https://community.chocolatey.org/packages/keepassxc

Note that there's no longer a 32-bit version.
The existing 2.6.6 version is the final one that supports 32-bit.


What to do with/about phantom child spans?

I'm seeing a lot of these and while these 3 reference /messages I don't think they are directly related though
0|api      | @google-cloud/trace-agent ERROR TraceApi#createChildSpan: [mysql@2.16.0:lib/Connection.js] Creating phantom child span [mysql-query] because root span [/messages] was already closed.
0|api      | @google-cloud/trace-agent ERROR TraceApi#createChildSpan: [http@8.12.0:http] Creating phantom child span [metadata.google.internal] because root span [/messages] was already closed.
0|api      | @google-cloud/trace-agent ERROR TraceApi#createChildSpan: [grpc@1.15.1:src/client.js] Creating phantom child span [grpc:/google.logging.v2.LoggingServiceV2/WriteLogEntries] because root span [/messages] was already closed.

This might be related to https://github.com/googleapis/cloud-trace-nodejs/issues/382 and thus also to https://github.com/googleapis/cloud-trace-nodejs/issues/794
but wanted to be sure.
That endpoint does make use of tails atm.
The logging thing happens through<EMAIL_ADDRESS>but I can probably update that if that helps with anything :D
If there isn't anything I can do for now until the referenced issues are solved, would there be a way to turn these messages of since they are massively spammy atm

I've fixed the part related to log levels in #880, though with regards to tail latencies -- I am still working on enabling child spans as tail latencies (as tracked in #794). Thanks for bringing this to our attention.

The log level change will probably help 👌 (which was the main issue, I could see y'all working on the tail thing which is stellar ofc ✨ )
If the core of this issue is tracked with #794 you can close this issue ✅

Ok, thanks. I'll go ahead and close this issue


Quick triage comment: always a good idea.


DEPLOY-686: Updated ACS chart version to 3.0.0

Did you mean 2.0.0 ( https://github.com/Alfresco/acs-deployment/releases/2.0.0 ) for ACS 6.1.0 ?
https://github.com/Alfresco/acs-deployment/blob/master/docs/helm-versioning.md

Did you mean 2.0.0 for ACS 6.1.0 ?
https://github.com/Alfresco/acs-deployment/blob/master/docs/helm-versioning.md

https://github.com/Alfresco/acs-deployment/releases/2.0.0
https://github.com/Alfresco/acs-deployment-aws/releases/2.0.0

@montgolfiere Unfortunately the 2.0.0 release contains Digital Workspace 1.0.0 and we need the 1.1.0 version (this version uses the Identity Service) and this is why we picked 3.0.0 that we found here: https://github.com/Alfresco/charts/blob/master/incubator/alfresco-content-services-3.0.0.tgz

Hmm - does anyone know which repo / tag / build creates these charts / versions ?
Is this still valid https://github.com/Alfresco/acs-deployment/blob/master/docs/helm-versioning.md ?

@montgolfiere The purpose of this issue was to pull in a version of ACS that uses ADW 1.1 so we have
a working configuration that uses IDS.   That's been done by including the ACS 3.0 chart.   If/when there's a 2.x.x chart that uses ADW 1.1 we can consider using that instead.   That probably would make more sense if the ACS 2.x.x chart includes ACS 6.1.1.


VCS install using https credentials in env variables fails for Pipfile

Issue description
Trying to install a package over https, from a private AWS CodeCommit repo, on an Ubuntu system.
Instead of using git credential.helper, we would like to inject credentials into our Pipfile using env variables.
So, in a bash terminal, we define USERNAME and PASSWORD (url-encoded).
Installing into an empty dir, directly from the command line, using these credentials, appears to work without issue:
pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"

This implies that the credentials and user permissions are correct.
However, it turns out the username and password end up in both Pipfile and Pipfile.lock...
That does not sound like a good idea.
Luckily, the docs for Injecting credentials into Pipfile via environment variables say:

Keep in mind that environment variables are expanded in runtime, leaving the entries in Pipfile or Pipfile.lock untouched. This is to avoid the accidental leakage of credentials in the source code.

However, if we try to install from a Pipfile into an otherwise empty dir, using the exact same url with same USERNAME and PASSWORD, we get a status 403 (Forbidden):
# ...
[packages]
mypackage = {editable = true, ref = "main", git =<EMAIL_ADDRESS># ...

yields
...
INFO:pip.subprocessor:fatal: unable to access 'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/':
The requested URL returned error: 403
ERROR:pip.subprocessor:git clone --filter=blob:none --quiet<EMAIL_ADDRESS>/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
exited with 128
...

Expected result
I expect installation from Pipfile to work without issue, just like installation from the command line.
Actual result
command line output
/tmp/pipenvtest $ pipenv install --verbose
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
INFO:pipenv.patched.pip._internal.operations.prepare:Collecting mypackage@
<EMAIL_ADDRESS>mazonaws.com/v1/repos/mypackage@main (from -r
/tmp/pipenv-2_t8zu99-requirements/pipenv-3cdgwj18-constraints.txt (line 2))
INFO:pipenv.patched.pip._internal.vcs.git:Cloning
<EMAIL_ADDRESS>naws.com/v1/repos/mypackage (to revision main) to
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet
<EMAIL_ADDRESS>onaws.com/v1/repos/mypackage'
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
INFO:pip.subprocessor:fatal: unable to access
'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/': The
requested URL returned error: 403
ERROR:pip.subprocessor:git clone --filter=blob:none --quiet
<EMAIL_ADDRESS>onaws.com/v1/repos/mypackage'
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with
128
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 455, in resolve
    results = resolver.resolve(constraints, check_supported_wheels=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 534, in collect_root_requirements
    reqs = list(
           ^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 294, in __init__
    super().__init__(
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 157, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 223, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 305, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 596, in _prepare_linked_requirement
    local_file = unpack_url(
                 ^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 157, in unpack_url
    unpack_vcs_link(link, location, verbosity=verbosity)
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 80, in unpack_vcs_link
    vcs_backend.unpack(location, url=hide_url(link.url), verbosity=verbosity)
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 608, in unpack
    self.obtain(location, url=url, verbosity=verbosity)
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 521, in obtain
    self.fetch_new(dest, url, rev_options, verbosity=verbosity)
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
git.py", line 276, in fetch_new
    self.run_command(
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 650, in run_command
    return call_subprocess(
           ^^^^^^^^^^^^^^^^
  File
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/util
s/subprocess.py", line 224, in call_subprocess
    raise error
pipenv.patched.pip._internal.exceptions.InstallationSubprocessError: git clone
--filter=blob:none --quiet
<EMAIL_ADDRESS>onaws.com/v1/repos/mypackage'
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with
128
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line
675, in <module>
    main()
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line
661, in main
    _main(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line
645, in _main
    resolve_packages(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line
612, in resolve_packages
    results, resolver = resolve(
                        ^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line
592, in resolve
    return resolve_deps(
           ^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 932, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 700, in actually_resolve_deps
    resolver.resolve()
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 457, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: [31m[1mERROR[0m: [33mgit clone
--filter=blob:none --quiet
<EMAIL_ADDRESS>onaws.com/v1/repos/mypackage'
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with
128[0m
✘ Locking Failed!
⠋ Locking packages...
Traceback (most recent call last):
  File "/home/me/.local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 653, in do_init
    do_lock(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 737, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate
first

create a private aws codecommit repo with a minimal pyproject.toml, as follows, and setup a user with git credentials (perhaps a private github repo would also work, but haven't tried that yet)[build-system]
requires = ["setuptools >= 61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.0.1"

in a bash terminal on the local system, create env variables USERNAME and PASSWORD (e.g. using read -s) with the corresponding values (url-encoded)
create an empty dir, locally, and use pipenv to install the repo, in editable mode (not sure if that matters), as in:pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"

observe that this installation succeeds

then

in the same terminal, create another empty dir
copy the Pipfile, generated above, into the new dir, and replace the actual username and password in the file by the corresponding env variables, as in:[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
mypackage = {editable = true, ref = "main", git =<EMAIL_ADDRESS>
[dev-packages]

[requires]
python_version = "3.11"

run pipenv install
observe the status 403 error

$ pipenv --support
Pipenv version: '2024.0.1'
Pipenv location: '/home/me/.local/lib/python3.11/site-packages/pipenv'
Python location: '/home/me/.pyenv/versions/3.11.6/bin/python3.11'
OS Name: 'posix'
User pip version: '24.0'
user Python installations found:
PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.11.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '6.5.0-41-generic',
 'platform_system': 'Linux',
 'platform_version': '#41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 '
                     '11:32:55 UTC 2',
 'python_full_version': '3.11.6',
 'python_version': '3.11',
 'sys_platform': 'linux'}

System environment variables:

SHELL
SESSION_MANAGER
QT_ACCESSIBILITY
PIPENV_VENV_IN_PROJECT
COLORTERM
PYENV_SHELL
XDG_CONFIG_DIRS
SSH_AGENT_LAUNCHER
XDG_MENU_PREFIX
GNOME_DESKTOP_SESSION_ID
LANGUAGE
LC_ADDRESS
GNOME_SHELL_SESSION_MODE
LC_NAME
SSH_AUTH_SOCK
GIT_PS1_SHOWDIRTYSTATE
XMODIFIERS
DESKTOP_SESSION
LC_MONETARY
GTK_MODULES
PWD
LOGNAME
XDG_SESSION_DESKTOP
XDG_SESSION_TYPE
SYSTEMD_EXEC_PID
XAUTHORITY
HOME
USERNAME
IM_CONFIG_PHASE
LC_PAPER
LANG
LS_COLORS
XDG_CURRENT_DESKTOP
VTE_VERSION
WAYLAND_DISPLAY
GNOME_TERMINAL_SCREEN
GNOME_SETUP_DISPLAY
LESSCLOSE
XDG_SESSION_CLASS
TERM
LC_IDENTIFICATION
LESSOPEN
USER
GNOME_TERMINAL_SERVICE
DISPLAY
SHLVL
LC_TELEPHONE
QT_IM_MODULE
LC_MEASUREMENT
PAPERSIZE
XDG_RUNTIME_DIR
PYENV_ROOT
LC_TIME
XDG_DATA_DIRS
PATH
GDMSESSION
DBUS_SESSION_BUS_ADDRESS
LC_NUMERIC
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

PIPENV_VENV_IN_PROJECT: 1

Debug–specific environment variables:

PATH: /home/me/.pyenv/versions/3.9.18/bin:/home/me/.pyenv/versions/3.8.18/bin:/home/me/.pyenv/versions/3.11.6/bin:/home/me/.pyenv/versions/3.8.13/bin:/home/me/.pyenv/shims:/home/me/.npm-global/bin:/home/me/.local/bin:/home/me/.pyenv/bin:/home/me/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
SHELL: /bin/bash
LANG: en_CA.UTF-8
PWD: /home/me

@matteius I've tried to reproduce this using one of my private github repos, with github username and fine-grained PAT as credentials. However, in this case pipenv already fails at replication step 3. mentioned above (non-editable).
For example, pip succeeds:
pip install<EMAIL_ADDRESS>
but pipenv, with the same command in the same terminal
pipenv install<EMAIL_ADDRESS>
fails with
...
Added my-private-repo to Pipfile's [packages] ...
✔ Installation Succeeded
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (13a959)...
[pipenv.exceptions.InstallError]: Collecting my-private-repo<EMAIL_ADDRESS>(from -r /tmp/pipenv-hhg4plkm-requirements/pipenv-vmlg4kcl-reqs.txt (line 1))
[pipenv.exceptions.InstallError]:   Cloning<EMAIL_ADDRESS>to /tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]: Running command git clone --filter=blob:none --quiet<EMAIL_ADDRESS>/tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]:   fatal: unable to access 'https://0888...b155/': Could not resolve host: 0888...b155
...
ERROR: Couldn't install package: {}
Package installation failed...

@dennisvang Could you test this report against this branch with I believe does better with the VCS env variables:  https://github.com/pypa/pipenv/pull/6242

Analysis of Issue #6195:
1. Problem Summary
The issue reports a discrepancy in Pipenv's handling of HTTPS credentials for VCS installations when specified via environment variables. While direct command line installation with embedded credentials works as expected, using the same URL format with environment variables in a Pipfile leads to a 403 Forbidden error during installation.
2. Comment Discussion Analysis

matteius seeks help setting up a private repo for testing, indicating an interest in addressing the issue.
dennisvang provides valuable insights by attempting to reproduce the issue with a private Github repository. They note that even direct command-line installation with embedded credentials fails in their case, suggesting a potential broader issue with Pipenv's VCS credential handling.
matteius suggests testing against a specific branch (PR #6242), hinting at potential improvements in VCS environment variable handling.

3. Proposed Resolution
The issue likely stems from Pipenv not correctly substituting environment variables within the Pipfile before passing the URL to the underlying VCS command. The solution involves ensuring proper variable expansion and possibly refining how credentials are handled during VCS installations.
Here's a detailed breakdown:

Variable Expansion: Pipenv should expand environment variables within the Pipfile's VCS URL before invoking the installation command. This ensures the actual username and password are passed to the VCS tool.
Credential Handling:  Investigate if the current approach of directly embedding credentials in the URL is the most secure and robust method.  Consider alternative approaches, like using a credential helper or dedicated environment variables for VCS credentials.
Regression Testing: Thoroughly test various VCS installation scenarios, including different VCS providers (GitHub, CodeCommit, etc.) and credential formats, to ensure the fix effectively addresses the issue and doesn't introduce regressions.

4. Potential Code Changes
Enhance pipenv/utils/project.py:
from pipenv.utils.shell import safe_expandvars

class Project:
	  # ... existing code ...

	  def get_vcs_deps(self, dev=False):
	      # ... existing code ...

	      # Expand environment variables in the URL.
	      for k, v in packages.items():
	          if is_vcs(v) or is_vcs(k):
	              if isinstance(v, dict):
	                  for key, value in v.items():
	                      if key in VCS_LIST:
	                          v[key] = safe_expandvars(value)

	      return packages or {}
	  # ... existing code ...

This modification expands environment variables within the VCS URL retrieved from the Pipfile before further processing.
5. Additional Steps/Investigations

Review PR #6242:  Analyze the changes in PR #6242 to understand its impact on VCS credential handling and whether it addresses this specific issue.
Explore Credential Helpers: Investigate if leveraging a VCS credential helper (like the Git Credential Manager) provides a more secure and flexible alternative to embedding credentials in URLs.
Security Considerations:  Thoroughly assess the security implications of the chosen credential handling approach, ensuring no sensitive information is inadvertently exposed or stored insecurely.

By addressing the variable expansion issue and potentially refining VCS credential management, Pipenv can ensure consistent and secure handling of private repositories, enhancing its usability and reliability for diverse projects and environments.

Could you check @dennisvang if https://github.com/pypa/pipenv/pull/6276 solves for this (I think it should).


failed to handle multiple devices for container. Skipping Filesystem stats

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG

Environment:

Cloud provider or hardware configuration:
AZURE
OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
Linux 3.10.0-514.2.2.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Version of Ansible (ansible --version):
ansible <IP_ADDRESS>

Kargo version (commit) (git rev-parse --short HEAD):
c2c334d
Network plugin used:
Flannel
Copy of your inventory file:
## Configure 'ip' variable to bind kubernetes services on a
## different ip than the default iface
node1 ansible_ssh_host=XX        ip=<IP_ADDRESS>
node2 ansible_ssh_host=XX   ip=<IP_ADDRESS>
node3 ansible_ssh_host=XX    ip=<IP_ADDRESS>
node4 ansible_ssh_host=XX    ip=<IP_ADDRESS>
## configure a bastion host if your nodes are not directly reachable
bastion ansible_ssh_host=x.x.x.x
[kube-master]
node1
node2
[etcd]
node1
node2
[kube-node]
node3
node4
[k8s-cluster:children]
kube-node
kube-master
Command used to invoke ansible:
Output of ansible run:

Nothing
Anything else do we need to know:

The cluster is new no modification.
I have this issues with kubelet:
E0328 21:59:31.366391    7283 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd with output stdout: 595672	/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd
, stderr: du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28462/task/28462/fd/4': No such file or directory
du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28462/task/28462/fdinfo/4': No such file or directory
du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28462/fd/4': No such file or directory
du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28462/fdinfo/4': No such file or directory

exit status 1, rootInodeErr: , extraDiskErr:
I0328 21:59:34.247568    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 21:59:34.247590    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 21:59:44.316774    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 21:59:44.316813    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 21:59:50.039604    7283 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/9ddecbf6-13fd-11e7-b7ff-000d3af44ea2-default-token-3fk97" (spec.Name: "default-token-3fk97") pod "9ddecbf6-13fd-11e7-b7ff-000d3af44ea2" (UID: "9ddecbf6-13fd-11e7-b7ff-000d3af44ea2").
I0328 21:59:54.474453    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 21:59:54.474474    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
E0328 22:00:04.151637    7283 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd with output stdout: 595672	/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd
, stderr: du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28843/task/28843/fd/4': No such file or directory
du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28843/task/28843/fdinfo/4': No such file or directory
du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28843/fd/4': No such file or directory
du: cannot access '/var/lib/docker/overlay/4c812f66192844591cbe50a740c1c5717490679c6d1ece499997fc00f54eb4bd/merged/proc/28843/fdinfo/4': No such file or directory
exit status 1, rootInodeErr: , extraDiskErr:
I0328 22:00:04.488010    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:04.488038    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:14.643745    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:14.643766    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:24.718877    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:24.718900    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:34.756636    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:34.756657    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:44.878780    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:44.878804    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:54.977996    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats
I0328 22:00:54.978010    7283 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats

Azure issue ? Its the first time i have this issue since i updated kargo

i used a old version of kargo like checkout : ee5f009
its working

etcd seem failing
2017-03-29 17:37:05.425593 W | etcdserver: apply entries took too long [13.151998ms for 1 entries]
2017-03-29 17:37:05.425667 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:07.488035 W | etcdserver: apply entries took too long [15.395283ms for 1 entries]
2017-03-29 17:37:07.488057 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:07.816239 W | etcdserver: apply entries took too long [28.691685ms for 1 entries]
2017-03-29 17:37:07.816274 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:11.613246 W | etcdserver: apply entries took too long [31.425189ms for 1 entries]
2017-03-29 17:37:11.613264 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:13.972708 W | etcdserver: apply entries took too long [15.45919ms for 1 entries]
2017-03-29 17:37:13.972975 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:17.785390 W | etcdserver: apply entries took too long [13.37581ms for 1 entries]
2017-03-29 17:37:17.785409 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:21.910395 W | etcdserver: apply entries took too long [19.405241ms for 1 entries]
2017-03-29 17:37:21.910463 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:24.113755 W | etcdserver: apply entries took too long [34.237306ms for 1 entries]
2017-03-29 17:37:24.113776 W | etcdserver: avoid queries with large range/delete range!
2017-03-29 17:37:26.129218 W | etcdserver: apply entries took too long [35.295047ms for 1 entries]
2017-03-29 17:37:26.129241 W | etcdserver: avoid queries with large range/delete range!

I updated to etcd:v3.1.5 now its working fine but :
2017-03-30 13:36:08.632549 W | etcdserver: server is likely overloaded
2017-03-30 13:36:09.648226 W | wal: sync duration of 1.015558455s, expected less than 1s
2017-03-30 13:36:09.648359 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 515.819967ms)
2017-03-30 13:36:09.648375 W | etcdserver: server is likely overloaded
2017-03-30 13:36:10.198314 W | etcdserver: apply entries took too long [190.546297ms for 1 entries]
2017-03-30 13:36:10.198335 W | etcdserver: avoid queries with large range/delete range!
2017-03-30 13:36:12.541582 W | wal: sync duration of 1.008510845s, expected less than 1s
2017-03-30 13:36:12.541665 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 754.661292ms)
2017-03-30 13:36:12.541673 W | etcdserver: server is likely overloaded
2017-03-30 13:45:35.387604 I | mvcc: store.index: compact 919
2017-03-30 13:45:35.389203 I | mvcc: finished scheduled compaction at 919 (took 1.224647ms)
are still presents


[Feature] add connector finish sink interface

What type of PR is this：

[ ] BugFix
[x] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #
https://github.com/StarRocks/starrocks/issues/21502
Problem Summary(Required) ：

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[ ] I have checked the version labels which the pr will be auto backported to target branch

[ ] 3.0
[ ] 2.5
[ ] 2.4
[ ] 2.3

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@f9cabc8). Click here to learn what that means.
Patch has no changes to coverable lines.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #21628   +/-   ##
=======================================
  Coverage        ?   60.30%
  Complexity      ?    30043
=======================================
  Files           ?     2142
  Lines           ?   171710
  Branches        ?    25750
=======================================
  Hits            ?   103554
  Misses          ?    58763
  Partials        ?     9393

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


App Scan Datamatrix with NHRN's

PL user story: #93 (see full overview for additional information)
Description:
Changes to the App and decoding in order to be able to scan additional Datamatrix codes which contain NHRN (National Healthcare Reimbursement Numbers:

The ability to scan and decode a Datamatrix code which has NHRN's included:

The rules for decoding and finding the Anchor could be:

The Datamatrix code could have up to 6 AI (Application Identifiers) included depending on the target market:
AI01 (GTIN)
AI21 (Serial Number)
AI10 (Batch Number)
AI17 (Expiry Date)
AI71x (NHRN - see table above)

Acceptance criteria's:
positive tests

Scan the following barcodes and the Datamatrix code should be decoded and the ePI shown.
Datamatrix code using  and encoded in the order 01+21+10+17+710+714

Datamatrix code using FNC1 and encoded in the order 01+21+10+17+710+714

Datamatrix code using FNC1 and encoded in the order 01+21+711+713+10+17

negative tests / error path testing

N/A

Process scenario:

N/A

Additional information:
The App scan and decodes the barcodes and the ePI is shown.

This was tested and works as expected.


add kaktus to "See also" section

Also added to awesome-choo. It's okey if you reject it here. :) Just wanted to share this amazing thing :)

cool!


Fail update early if dirty

Currently, user will be prompted with the previous answers. If the repository is dirty, update will be aborted. To avoid being prompt for nothing, the update should fail before prompting the user.

I've been looking at this and it's not as easy as it seems right now:

To know if we're updating, first we have to make some guesses based on the results of make_config: https://github.com/pykong/copier/blob/a8e684736b74a4deb04cd0903d50444c8e21149d/copier/main.py#L124-L133
make_config is in charge of asking the user.
Doing additional calls to make_config clones the source more times, which slows down the thing like a lot.

Basically this is too hard to fix as the code is right now. I'll change milestone here because this would be much easier after fixing #110.

This was fixed by the refactor 🎉


[SPARK-37445][BUILD] Upgrade hadoop profile to hadoop-3 and change GA test name to hadoop3.3

What changes were proposed in this pull request?
Upgrade hadoop profile to hadoop-3 since we support hadoop-3.3 as default now.
In current project, deps's path is still hadoop-3.2, it's not correct.
In this pr also change GA test name to hadoop-3.3
Why are the changes needed?
Upgrade hadoop profile
Does this PR introduce any user-facing change?
In this pr, we change hadoop profile form hadoop-3.2 to hadoop-3, and hadoop-3 is default hadoop profile.
User can build other lower version by -Dhadoop.version=3.2.1 such as
./build/mvn -Pyarn -Dhadoop.version=3.2.1 -DskipTests clean package

If user want to built hadoop 2 related version. you can run
./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.8.3 -DskipTests clean package

How was this patch tested?
Not need

Test build #145539 has started for PR 34689 at commit 6a446eb.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50011/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50011/

Test build #145539 has finished for PR 34689 at commit 6a446eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145539/

Test build #145567 has started for PR 34689 at commit 7d8e8b4.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50039/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50039/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50039/

Test build #145606 has started for PR 34689 at commit 3e132f3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50080/

Also ping @cloud-fan

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50080/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50080/

how about the file name of the final release binaries?

how about the file name of the final release binaries?

In make-distribution.sh , package name is passed by option --name
if [ "$MAKE_TGZ" == "true" ]; then
  TARDIR_NAME=spark-$VERSION-bin-$NAME
  TARDIR="$SPARK_HOME/$TARDIR_NAME"
  rm -rf "$TARDIR"
  cp -r "$DISTDIR" "$TARDIR"
  tar czf "spark-$VERSION-bin-$NAME.tgz" -C "$SPARK_HOME" "$TARDIR_NAME"
  rm -rf "$TARDIR"
fi

In release-build.sh, it use parameter in BINARY_PKGS_ARGS

  if [[ $PUBLISH_SCALA_2_12 = 1 ]]; then
    echo "Packages to build: ${!BINARY_PKGS_ARGS[@]}"
    for key in ${!BINARY_PKGS_ARGS[@]}; do
      args=${BINARY_PKGS_ARGS[$key]}
      extra=${BINARY_PKGS_EXTRA[$key]}
      if ! make_binary_release "$key" "$SCALA_2_12_PROFILES $args" "$extra" "2.12"; then
        error "Failed to build $key package. Check logs for details."
      fi
    done
  fi

In this PR I have change the BINARY_PKGS_ARGS . So the release package name should be hadoop3.3
I have no committer authority, so can't do an end to end test.

how about the file name of the final release binaries?

Also ping @vanzin , since this part of code is changed by him.

Test build #145606 has finished for PR 34689 at commit 3e132f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145606/

LGTM, please fix the conflicts.

LGTM, please fix the conflicts.

Done

Test build #145622 has started for PR 34689 at commit d3ad6e7.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50094/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50094/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50094/

Test build #145622 has finished for PR 34689 at commit d3ad6e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnclosedCommentProcessor(

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145622/

thanks, merging to master!

Guys, im gonna revert this - this fix is incomplete, and we should keep the name same as hadoop3 instead of hadoop3.3.

cc @dongjoon-hyun too FYI

Thank you for pinging me, @HyukjinKwon .

Guys, im gonna revert this - this fix is incomplete, and we should keep the name same as hadoop3 instead of hadoop3.3.

So what should I do next? raise a new pr to change all to hadoop3? or left it to @sunchao

Yes, renaming profiles and release file names were discussed at https://mail-archives.apache.org/mod_mbox/spark-dev/202106.mbox/<CAF6oT1dOVc86zPFiCEcPtq6CEDknKxu6bzGPm5NZ%3Dms706kfxg%40mail.gmail.com>. I don't mind who works on that but let's make sure do it in one go.

Ah, I miss the email. I will rase a new one. Thanks for your review.

So what should I do next? raise a new pr to change all to hadoop3? or left it to @sunchao

Feel free to take over @AngersZhuuuu . I can help on reviewing.

So what should I do next? raise a new pr to change all to hadoop3? or left it to @sunchao

Feel free to take over @AngersZhuuuu . I can help on reviewing.

Thanks a lot.


Changing a dynamically created plist's properties

Hi, great work, very useful and clever, :+1:
I'm using a dynamically created plist, and I wanted to know what happens if I delete a property from the PlistModel subclass, add new ones, or add one with the same name as a previously deleted one. According to my tests, all of the deleted properties still hold their previous values, so if you delete one and return it, it will hold its previous value. Also, if you add new properties, it doesn't interfere with the old ones (e.g. reset them), which is the desired behavior for my use case. Just wanted to make sure this is indeed the regular behavior, for when I'll want to add properties to the next version of my app.
Thanks, Amitai

Hi @amitai-hoze,
Let's try a couple things:

Just to confirm, this is definitely a dynamic plist and there is no matching plist included in the bundle?

If you're running in the simulator and rerunning the app from Xcode, sometimes the sigkill doesn't call dealloc and your values might not persisting.

If these don't work, would you be able to post come code?  It's probably not feasible if you're using it in a few different places, but if there's some that you have that I can use to replicate the problem, it would help me to figure out what the problem is.

Best,
Logan

Oh sorry, maybe my question wasn't clear enough..
There's no problem with the class, it's working great. I just wanted to find out how the class handles situations where properties were deleted or added to the PlistModel subclass .h file (that creates the plist file dynamically, like DynamicModel in your Xcode project).
Let's say for instance I've got this PlistModel subclass:
@interface DynamicModel : PlistModel

@property (strong, nonatomic) NSString * name;
@property int counter;

@end

The code that writes to the class looks like this:
[DynamicModel<EMAIL_ADDRESS>= someUserInputString;
[DynamicModel<EMAIL_ADDRESS>
I publish the app in the app-store, users are using it and all of them have name and counter saved values. Now I decide to add another property like so:
@interface DynamicModel : PlistModel

@property (strong, nonatomic) NSString * name;
@property (strong, nonatomic) NSString * newStringProperty;
@property int counter;

@end

are the original name and counter values guaranteed to stay the same?
And what happens if I delete the newStringProperty from the DynamicModel.h file, publish the app again, and then put it back and publish, will it have a Null value or the value saved from the last time it was saved?
In short, what is the behavior when the DynamicModel.h file changes between versions of the app (adding properties, deleting properties).
My tests indicate that all the former values are always preserved and never get reset, which is the desired behavior (in my use case at least). I just wanted to make sure I can rely on it.
Thanks,
Amitai

Ok @amitai-hoze, I understand the question much better!
So under the covers, the class is really just designating a file in the directory.  For your example above, this file is named DynamicModel.  When this is updated, the file is completely overwritten.  As far as I know, there isn't a way to edit only parts of the file, the way you can with something like a SQL database. I'll address some of your concerns here:
#1. Add a property that didn't exist before.
This will be nil when you load it from the old version.
#2. Remove a property that existed before.
This value will still exist in a backing dictionary when the plist is loaded from the directory.  This won't affect any of the other properties and after the first save, this will persist in the key value store until it is overwritten to nil.  If this isn't desired behavior, perhaps I should consider a good way to offer this as an option.  The reason this happens is sometimes values are accessed by key only and don't have an associated property.  You wouldn't want these to be overwritten in general.
In summation, the basic implementation is an underlying key value store, so anything added will be there until you remove it.  If you add new things, they'll be added to the rest, if you remove things there values will still exist as a key value pair until it is removed manually.
I also wanted to mention one thing about your use in the example above.  I'm not sure if this is how you're implementing it in project, but I wanted to mention it briefly.  This code:
[DynamicModel<EMAIL_ADDRESS>= someUserInputString;
[DynamicModel<EMAIL_ADDRESS>
Would run more efficiently if it were run like this:
DynamicModel *dynamicModel = [DynamicModel plistNamed:@"DynamicModel"];
dynamicModel.name = someUserInputString;
dynamicModel.counter++;

Because it prevents loading the data from the directory unnecessarily.  Normally this isn't an extremely expensive operation, but I wanted to mention it :)
Are there any questions / issues I missed?
Best,
Logan

@LoganWright, thanks a lot, your answer cleared up all my concerns :relieved:
Regarding your comment on accessing the properties more efficiently, you're right, I wrote it like that for brevity, but good comment anyways.
Thanks,
Amitai

Glad to hear it :) Good luck with the rest of your project!


Registered Services throws NullReferenceException when service is added

Describe the bug
When adding a service to the Registered Services a NullReferenceException is thrown by the inspector because adding the entry does not define the actual type of service.
To reproduce
Steps to reproduce the behavior:

Add SceneTransitionService, which currently is the only one

Expected behavior
No error
Your Setup (please complete the following information)

Unity Version 2018.4.6f1
MRTK Version mrtk_development

Target Platform (please complete the following information)

Editor

i believe this was fixed in 2.1.0 or 2.2.0. i am not experiencing it in the current development branch (and was, in fact, testing this scenario)


Update Gradle to 8.4,Update AGP to 8.1.2,Update kotlin to 1.9.10

@Kottsone  @jeanlucbr  Is anybody there?

@Kottsone @jeanlucbr @ggfan is anybody there?


[Question] how would you see setting attributes that are related to each other?

Apologies if this is a dense question, but in this example I want the name and code to be related - e.g. the code should be the lowercase value of name - but name is generated using faker (or chancejs, etc).
export class ChannelFactory extends Factory<Channel> {

  entity = Channel;

  attrs = {
    name: () => `Web-${faker.datatype.number({ min: 1000, max: 9999 })}`,
    code: // should be the name .toLowerCase(),
    ...
  };
}

Do you have a suggested pattern in terms of how to handle this situation?
Thanks for the work so far on this tool, seems useful!

Hi Kevin, I'm really happy this lib is useful to someone else than me. I have some plans to extend it to work outside of typeorm but I'm giving myself some time to think about it
You can achieve what you want with what I called LazyAttribute. You can apply a function which get as an argument the instance to be created. In your case, it means you could access your name property !
There is an example in the readme about LazyAttribute

Unless there is a bug with the implementation, I'll close this issue

Thanks so much for taking the time to make the tool.
I spent some time looking at some of the other options and like your approach. Clearly I should have spent some more time reading your documentation instead :).
I will take a look at the LazyAttribute, it sounds like what I want!
One other unrelated note (I can make an issue if you want for tracking?) - I think it might be useful to add more information about HOW the persistence works with TypeORM.  You have

The adapter allows you to persist your data. If you want to save your data in a database via typeorm, you can use the TypeormAdapter. Default Adapter is ObjectAdapter and does not persist anything. You can create your own adapter to persist your data the way you want.

But it isn't clear to me from this documentation alone how this actually works, or what the intended work flow would be here. I will likely use this for seeding some basic data in lower dev / sandbox environments, or potentially for end-to-end testing, and I can always try to open a PR with more details if I get to it.
For now, thanks again, I'll close out this issue!

If you need more information, you can have a look at typeorm
This project does not aim to document typeorm features.
But you can also use this lib without using typeorm !
It is quite common when using factories, seeding etc... to have data being persisted in a database, a file or whatever.
Factory uses a method called save to persist data. If no adapter is provided, this method does... almost nothing. Adapters, like the typeorm one, allow you to use typeorm features and save the data in your database. What is not written in my documentation is how typeorm should be setup, how you open a connection to your database etc... This should be covered on their documentation though.
If you need any kind of custom logic in your save method, you could write your own adapter to suit your needs.


Adjustment on directory creation to work with Ubuntu distribution

As my report in issue #31. It looks like the current module doesn't work with Ubuntu. Also, on ubuntu, the default configuration file for etc should be /etc/default/etcd as following systemd file
cat /lib/systemd/system/etcd.service
[Unit]
Description=etcd - highly-available key value store
Documentation=https://github.com/coreos/etcd
Documentation=man:etcd
After=network.target
Wants=network-online.target

[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"
ExecStart=/usr/bin/etcd $DAEMON_ARGS
Restart=on-abnormal
#RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd2.service

Are you sure you want to create /etc/etcd and not /etc/default/etcd?
Also, isn't this dir supposed to be created by the package?
Anyway, let me know if you actually want /etc/etcd and I will merge it

Hi,
Sorry for late response.  /etc/default/etcd is automatically created by the package itself, but /etc/etcd not. I am not sure whether /etc/etcd is being used by etcd pacakge in ubuntu, but with this code, the puppet module is trying to create a file in non-existing directory.

Sorry for the delay


Add new firebase_core plugin and update firebase_database to use it

Fixes flutter/flutter#13108
Android side is still WIP

Landing this as two PRs, firebase_core changes need to land first so I can make firebase_database depend on firebase_core version 0.0.1. Splitting firebase_database out into a second PR.


Chore: Make Black start at 0 in AnsiColor

With these changes, we can get rid of the fromIndex() method. Wdyt?

why not


Feature: client auto-scaling based on planned allocations instead of actual CPU usage

Currently the cluster doesn't scale up even when all clients are fully allocated, leaving no room for new allocations. Also see https://developer.hashicorp.com/nomad/tools/autoscaling/agent/high_availability.

We could run Prometheus to capture the metrics, then use the stackdriver exporter to send those to GCP, then create scaling rules on those metrics.
https://github.com/prometheus-community/stackdriver_exporter

Working on adding Prometheus in #15

Fixed by using open telemetry collector, GCP managed prometheus and metrics-based auto scaler config.


Fix fetch_provisioning_profile when output_path is provided as a match option

Checklist

[x] I've run bundle exec rspec from the root directory to see all new and existing tests pass
[x] I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
[x] I see several green ci/circleci builds in the "All checks have passed" section of my PR (connect CircleCI to GitHub if not)
[x] I've read the Contribution Guidelines
[x] I've updated the documentation if necessary.
[x] I've added or updated relevant unit tests.

Motivation and Context

Resolves https://github.com/fastlane/fastlane/issues/21945
Description

Given some investigation and subsequent conversation, this PR proposes that the line of code causing #21945 can simply be removed. It should not be necessary to run FastlaneCore::ProvisioningProfile.install a second time when output_path is provided as a match option.
Testing Steps

For my own testing, I used this branch of fastlane to execute the same CI workflow (from a private project) that was failing on 2.220.0 with the error described in #21945. The match actions for all builds, as well as the subsequent gym actions, all ran successfully with no observable side effects.
match(
    output_path: "/tmp/match_certs", # must include, this creates the condition that this PR fixes
    # everything below is merge suggestion, use whatever you consider to be a standard set of match params
    git_url: YOUR_MATCH_REPO_URL,
    type: "adhoc",
    app_identifier: YOUR_APP_IDENTIFIER,
    force_for_new_devices: true,
    include_all_certificates: true,
    force_for_new_certificates: true,
    shallow_clone: true,
    readonly: true,
)

How long before this fix is released? It's fixing a pretty serious breaking bug.
We are using fastlane only to resign, so we don't have a gemfile to pin to the previous version.

Hi guys, anyone here? I still got the issue.
I'm running it on self-hosted (MacBook M1 Pro) github action.

Same here, I'd really appreciate some update on the potential release date. Thanks!


How to prevent data queues to become too long?

By pushing to flow which does not process data fast, it is possible, that it's queue gets too long. We need to be able to handle such situations.

The inverse of the problem, when a flow is drained, because we try to pull it too fast. These problems can occur by unsynchronized flows.

Need some examples about unsynced flows. The simplest example is merging 2 pushed flows, but these flows will have a different push interval. After a while the rapidly pushed flow will become saturated. I need probably a threshold and trigger an event then.

Hmm it is possible to add multiple examples and start them only if we add CLI params. E.g. npm run examples --start "advanced/long-queue"

I am not sure about how to implement this feature. I mean should it be built in to the flow or to the pump? I think the flow should contain only a size() method, and the pump should check the size() and if it is too big, that should trigger a threshold-reached event.

I am not sure whether I should emit a size event, or add a threshold event. Probably a refreshed or refresh event can be used to measure size, but adding a threshold might be better.