[Feature] Gracefully roll workloads with only 1 replica

Tell us about your request
We have a few Pods in our system which we can only run 1 of (and that we preferably don't want to be running 0 of).
Depending on how you implement #452, when a Node is Terminated and gracefully drained, etc, presumably the Controller will just Delete all the Pods and let Kubernetes schedule replacements, or, if it would violate the PDB, do nothing. Rather than just deleting these, it would be nice if there was an option to instead do a rollout restart so that a new pod is brought up (and becomes Ready) before the old one is Terminated.
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
The impact of not doing it this way is an interruption to our Pods which are replicas: 1. Of course the real fix is to solve that, but that is not always possible e.g tech-debt, licensing, etc.
Are you currently working around this issue?
We currently use cluster-autoscaler, node-termination-handler and our own custom controller (written with kubebuilder) to detect the Taints being added by node-termination-handler and then we rollout restart any Pods which are single replicas and have a PDB.
Additional context
We don't currently use Karpenter, we're holding off until a few features we need land in/around 0.3, but we wanted to flag this as a use-case.
Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Taking a look through https://github.com/kubernetes/kubectl/blob/652881798563c00c1895ded6ced819030bfaa4d7/pkg/cmd/rollout/rollout_restart.go#L134, I think we could support this by checking owner reference of pods, and then patching in a change to the corresponding statefulset, daemonset, or deployment after cordoning the node. Instead of evicting the pods ourselves, we could simply rely on the controller to execute the rollout to gracefully ensure uptime of the pods.
This is a great suggestion and I totally see the value here. @njtran, do you think you can scope this into #452?

Yeah that's exactly how we do it. We check the owner and then update spec.template.metadata with a restarted-at annotation (rather than pulling in kubectl rollout).

Alternatively if you can't or don't want to handle this logic directly in karpenter, all we need is an event we can subscribe to so that we can do this ourselves.
The current Taint based approach of node-termination-handler is perfectly suitable, since we can watch for that, but we're happy to adapt to whatever you come up with (e.g webhook, etc).

Alternatively if you can't or don't want to handle this logic directly in karpenter
I think we should handle this.

all we need is an event we can subscribe to so that we can do this ourselves.
I'm trying to think about potential extension points. Right now, you could potentially annotate pods with do-not-evict and then handle the drain yourself when a node.metadata.deletionTimestamp is set. It strikes me that it might be a bit annoying for you to label every single pod with do-no-evict to achieve this behavior. I wonder if it would make sense to support an additional node label (or annotation?) of do-not-evict, and we would simply rely on a third party to do the eviction.

Thinking a little further on this, I'd probably want to constrain this behavior in some way. It would be surprising to trigger a full rollout to any deployment that happens to have a pod on a terminating node, and also rollout all the pods on happily running nodes.
I could see doing this with a pod annotation rollout-on-evict, or potentially constraining this behavior to only execute if the deployment/statefulset/daemonset has one replica. This makes less sense in the daemonset case, since you'll likely have many of these that won't need the rollout.

Another approach is to use termination grace period. When the evict is triggered, you could use a combination of PreStop[https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks] and TerminationGracePeriod to clean up, but this won't allow new connections to be made, since the Endpoint resources will be removed as soon as eviction is triggered.

Also -- can you describe the case where this is useful for daemonsets? I can see deployments/statefulsets with 1 replica.

It's definitely not useful for DaemonSets, nor is it really useful for anything with more than 1 replica, since as you said, you don't want everything rolling because 1 Node got terminated (and in the case of multiple Nodes terminating at once, you'll end up with multiple PATCH/PUTs which can get a bit jank).
Unfortunately we haven't open-sourced the code for our Controller (yet), otherwise I'd just link that so you can see the logic. But we only handle Deployments with replicas: 1 and when there's a matching PodDisruptionBudget (since otherwise it's a race with the node-termination-handler draining the Node and deleting the Pod anyway).
The annotation rollout-on-evict would be useful. We haven't implemented such logic since all our replicas: 1 Deployments benefit from being handled this way, but adding that annotation to our Deployments which require this functionality wouldn't be a problem for us (and I can see cases where people very explicitly don't want this to happen).

Although I think you understand it, for clarification the sequence of events in our current configuration is this:

cluster-autoscaler or a user starts a scale down
node-termination-handler receives the termination event
node-termination-handler cordons the node (adds the cordon taint)
node-termination-handler taints the node

If true, nodes will be tainted when an interruption event occurs. Currently used taint keys are aws-node-termination-handler/scheduled-maintenance, aws-node-termination-handler/spot-itn, aws-node-termination-handler/asg-lifecycle-termination and aws-node-termination-handler/rebalance-recommendation

Simultaneously:
5a. node-termination-handler starts deleting Pods (but any replicas: 1 pods with a PodDisruptionBudget cannot be deleted since it would violate the PDB and so node-termination-handler complains about these in its logs)
5b. our node-evacuator controller (which was watching for changes to the Nodes objects) sees one of the aws-node-termination-handler taints added to the Node and does a rollout restart (via adding a restarted-at label) of any replicas: 1 Pods with a matching PDB.

The Node is eventually empty (either from node-termination-handler deleting Pods or our Controller PATCHing their owner and causing a rollout restart and the Node is deleted and the Instance is Terminated

To further complicate (or maybe reduce the complexity of) this, Kubernetes has built in Graceful Termination now: https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/

Hey @mattparkes , I think that this is probably in the purview of another tool in the ecosystem. It should be straightforward to implement a custom controller that integrates w/ karpenter's termination workflows.

Yeah we can totally do that. As I said in my initial post, that's currently how we do it:

Are you currently working around this issue?
We currently use cluster-autoscaler, node-termination-handler and our own custom controller (written with kubebuilder) to detect the Taints being added by node-termination-handler and then we rollout restart any Pods which are single replicas and have a PDB.

However I do think that is definitely outside the reach of a lot of Amazon's customers, so it would be awesome if that was something you could provide at some point in the future, either as another tool or as a "plugin" for karpenter.
Can you give me any suggestions for the most suitable event that we should use to ascertain that karpenter is starting to terminate the Node? We currently use the taints added by node-terminationa-handler but skimming the code I don't see something similar in Karpenter?

For karpenter, node.deletionTimestamp will be set once karpenter starts to finalize the node.

Hello,
Two years later, I would like to know what's the most straight-forward way for achieving such a rollout in Karpenter's current version (0.29). Were some features implemented, that enable graceful rollout of deployments with 1 replica, e.g during consolidation or Spot Interruption ?
Thanks in advance for your help !

Hi @mattparkes @ellistarn
I have a similar situation where I have an application with a single replica and I can't perform rebalancing without downtime.
Could you share maybe the most up-to-date solution for the issue? Like share your custom contoller or any other examples?

@Frodox I followed @ellistarn's advice https://github.com/aws/karpenter-provider-aws/issues/500#issuecomment-993009615 and wrote a custom controller using Kubebuilder.  My initial solution, albeit clunky, works, but could have probably been done better.  It also was my first attempt with using Kubebuilder, or any similar framework; albeit I did have some client-go experience).
To get this to work, single replica deployments would require an associated blocking (minAvailable = 1, or maxUnavailable = 0) PodDisruptionBudget to be deployed, and a Nodepool having a terminationGracePeriod specified, I used 48h.
My NodeReconciler is constructed with a copy of the manager's Config (*rest.Config), and a command line parameter value for eviction delay which defaults to 30s (will get back to this).
When the controller is setup, I use a predicate that only returns true for updates to nodes labeled with "karpenter.sh/nodepool" and where the node's DeletionTimestamp transitions from its zero value to a non-zero value -- when the APIServer sets the node's DeletionTimestamp.
So as per the documentation, when a Karpenter node is deleted, the finalizer will block node deletion, yet the APIServer will set the DeletionTimestamp and Karpenter's node termination controller takes over, taints the node, then uses the K8s Eviction API to evict Pods as documented.
My simple solution is to wait an arbitrary, configurable, period of time (defaults to 30s), to allow Karpenter to evict all the Pods it's going to evict.  After this configurable period of time, the Pods left, that we care about, are PDB blocked Pods, but there could also be Pods remaining that are protected by a PDB limiting the disruption rate.
Using the NodeReconciler's config, the Reconcile function creates a clientset, and

gets the list of Pods on this node, then for any Pod that is owned by an apps/v1 ReplicaSet having replicas set to 1, that is owned by an apps/v1 Deployment.  The Pods' namespace and ReplicaSet owner's name are marked down.
for any deployment information marked down, these deployments are strategically-merged patched, adding/updating a PodTemplateSpec annotation that causing the rollout restart.

Each updated deployment goes through it's rollout restart and gracefully terminates the old replica on the node.  Once all such Pods on the node have exited, the node termination controller continues to reclaim the node.

@marc-ostrow thank you very much for the detailed answer!
Could you explain why you used a custom controller instead of an operator? What is the difference for this task?
I didn't quite catch the moment - does Karpenter's Termination Controller and your "NodeReconciler" run simultaneously? Or how is it ensured that NodeReconciler runs before/after/during Karpenter's Termination Controller?

To get this to work, single replica deployments would require an associated blocking (minAvailable = 1, or maxUnavailable = 0) PodDisruptionBudget to be deployed

We are talking about the situation where replicas: 1, right? But if we set pdb.minAvailable=1 / maxUnavailable=0, then the pod's pdb.status.disruptionsAllowed = 0 and Karpenter won't even start consolidation on this node with an error.

DisruptionBlocked node/ip-... Cannot disrupt Node: PDB "app-ns/app-pdb" prevents pod evictions

I also saw this in the documentation https://github.com/aws/karpenter-provider-aws/blob/main/designs/consolidation.md#pods-that-prevent-consolidation
I don't understand how consolidation starts for you. How did you manage to do this?

@Frodox

Could you explain why you used a custom controller instead of an operator? What is the difference for this task?

It's my understanding that an operator defines at least one CRD and one or more controllers, whereas I just needed to watch v1/Node's for their DeletionTimestamp being set.  I had run kubebuilder init specifying a company domain, a project name, and provided my gitlab repo name.  Then when I went to create the api with kubebuilder create api, I specified group core, version v1, and kind Node; and then answered the questions: 1) Create Resource [y/n] with "n" for no, and 2) Create Controller [y/n] with "y" for yes.

I didn't quite catch the moment - does Karpenter's Termination Controller and your "NodeReconciler" run simultaneously? Or how is it ensured that NodeReconciler runs before/after/during Karpenter's Termination Controller?

The short answer is "yes, independently, and during, by design.  It is assumed that the NodePools' terminationGracePeriod is longer than the time it takes to trigger the rollout-node-controller, for it to make the decision to update/patch the appropriate apps/v1 Deployments, and for possibly new node(s) to be scaled out and become ready, and for the terminationGracePeriods of the Pods terminating due to the rollout restart.   I'm defining my NodePools with a 48 hour terminationGracePeriod so this never would happen.

We are talking about the situation where replicas: 1, right? But if we set pdb.minAvailable=1 / maxUnavailable=0, then the pod's pdb.status.disruptionsAllowed = 0 and Karpenter won't even start consolidation on this node with an error.

Yes, for me, Pods owned by an apps/v1 ReplicaSet with spec.replicas set to 1, or not defined (?? which should never happen, but who's default value is 1); being owned by an apps/v1 Deployment. If the NodePool specifies a terminationGracePeriod (I am using 48h), then voluntary node disruption (consolidation), does, indeed, take place, but PDB blocked Pods are not evicted as the Eviction API returns an error indicated that they are PDB blocked.  So the Termination Controller will wait the specified NodePool terminationGracePeriod before forcefully killing the Pods and reclaiming the node.
However, nodes that have pods with a do-not-evict annotation are not considered for consolidation, as per the Troubleshooting documentation.

I also saw this in the documentation https://github.com/aws/karpenter-provider-aws/blob/main/designs/consolidation.md#pods-that-prevent-consolidation

Well, I know blocking PDBs with NodePool terminationGracePeriod set works, as this is how Karpenter 1.0.6 is working today.  I've already mentioned the Troubleshooting section on do-not-evict. And I'm not sure about the last point about not being able to move a Pod due to scheduling constraints...  I'd have to setup a test case, and assuming such a Pod is of a single replica deployment, the rollout-node-controller would perform a rollout-restart, which would cause a similarly specified Pod to be provisioned, and because the node was tainted, Karpenter would be forced to bring up yet another node meeting the Pods' scheduling constraints.

I don't understand how consolidation starts for you. How did you manage to do this?

I caused drift in the NodePool requirements: I had had minimum instance size requirements set to 2xlarge when a large instance size would have been more cost effective, then updated the requirements allowing large instance sizes.  I also disrupted the node simply by submitting a delete request for the node.

@Frodox Okay, I'm updating my rollout-node-controller to not use the filter so either on controller restart, or on leader election change, all nodes are looked at and discerned only if it's a karpenter node, and the DeletionTimestamp has a non-zero value.  Also, patching the deployment with an PodTemplateSpec annotation being my custom domain /restartedAt with a value set to the DeletionTimestamp of the node, so the deployment update becomes idempotent.

@marc-ostrow hmm, thank you for sharing your thoughts and insights!
Could you share your code for rollout-node-controller maybe in some way?


What's the relation between batch_sampler and real batchsize?

Hello, If I train a network with 2 gpus, the real batchsize is batch_sampler? or 2*batch_sampler?
what about the learning rate?
Thank you

the total batchsize = batchsize * gpu_num @LingyvKong


Publish MQTT Message on Motion ON and OFF

Is your feature request related to a problem? Please describe.
This is an expansion from https://github.com/seydx/homebridge-camera-ui/issues/425, we can now get MQTT messages about motion events (thank you!), but only at the conclusion of the recording/motion.
It'd be very useful to get motion events posted to MQTT on motion start. This would allow the Camera UI motion detection to be used to automate other things via MQTT (i.e. lights).
Describe the solution you'd like
For Motion ON and Motion OFF events to be able to be posted to an MQTT topic.
i.e. to be able to use these motion events:
Describe alternatives you've considered
It might be possible to do this via HomeKit automations using the associated motion sensor, but I'd rather go directly to MQTT.
Additional context
These Motion events are the ones I'd like to have available via MQTT.

Added in v5.0.23-beta.0

MQTT: When motion is detected, two MQTT messages are now published on following topics:

camera.ui/notifications: Contains all notifications AFTER motion has been detected AND recorded.
camera.ui/motion (can be changed in the interface): Contains motion event (before something is recorded).

v5.0.24 is online


Problem with Leopard

I tried the latest version (0.4.2) with Leopard (10.5.8). It seemed to open, but nothing ever appeared in the menu bar and disks would still mount.
Version 0.4.1 does still work.
Very useful utility. Has been quite handy recovering a damaged USB flash drive. Thanks.

Looks like this might be fixable by doing what Sparkle did:
https://github.com/sparkle-project/Sparkle/commit/d138f64bd6b6a6cc7c20ae7ba683ab81a49e6bd6

I tested briefly on 10.5 but the only system I have access to is 32-bit. However, I just released an 0.5 version, and for that I disabled the 64-bit version from running on 10.5, so this issue should be resolved.


Add TPCC QSR.

This adds Ternary Point Calculus relations to the library. This required the addition of a triadic abstract class.
See http://www.sfbtr8.spatial-cognition.de/project/r3/QualitativeCalculi/TPCC/index.html for details on TPCC.

Nice. Thanks. Will test it when I get a chance. Had a look on the documentation, which reference shall I use in the documentation (there are 4-5 papers in their references list).
Only comment/suggestion I have is that I would have defined the functions in qsr_lib/src/qsrlib_qsrs/qsr_tpcc.py as class @staticmethod with a protected namespace, but your call if you want them exposed publicly and also they will appear I think in the docs too even without a docstring.

Yes, it was messy :-) I have tidied it up.

Nice thanks. I will test it when I get a chance.

Had a quick test. What should the output of the following data be?
o1 = [Object_State(name="o1", timestamp=0, x=0., y=0., xsize=5., ysize=8.)]
o2 = [Object_State(name="o2", timestamp=0, x=5., y=0., xsize=5., ysize=8.)]
o3 = [Object_State(name="o3", timestamp=0, x=5., y=0., xsize=5., ysize=8.)]

for  o1, o2, o3?
Let me make a pull request to your branch firstly.

Is object_distance supposed to be from the relatum or from the origin? See this line.

Scrap previous comment. It should be from the relatum. But I think your if-statement seems not right. Shouldn't it be:
if object_distance == 0:

(can also use q-factor in this case, i.e. if object_distance <= 0.

Yes, but I made a mistake in the comparison after that. Should be 'sam'.

lol. just as I was writing it :P

What about the following:
        o1 = [Object_State(name="o1", timestamp=0, x=0., y=0., xsize=5., ysize=8.)]
        o2 = [Object_State(name="o2", timestamp=0, x=5., y=0., xsize=5., ysize=8.)]

it gives me always cbl for any of the three following values of o3.
o3 = [Object_State(name="o3", timestamp=0, x=6., y=0., xsize=5., ysize=8.)]
o3 = [Object_State(name="o3", timestamp=0, x=5., y=1., xsize=5., ysize=8.)]
o3 = [Object_State(name="o3", timestamp=0, x=6., y=1., xsize=5., ysize=8.)]
o3 = [Object_State(name="o3", timestamp=0, x=4., y=0., xsize=5., ysize=8.)]
o3 = [Object_State(name="o3", timestamp=0, x=4., y=-1., xsize=5., ysize=8.)]

But got a dbl for
o3 = [Object_State(name="o3", timestamp=0, x=50., y=50., xsize=5., ysize=8.)]

I made a mess when "doing a quick tidy up" to the code earlier, latest commit should fix it.

For o1, o2 always the same and as follows:
o1 = [Object_State(name="o1", timestamp=0, x=10., y=10., xsize=5., ysize=8.)]
o2 = [Object_State(name="o2", timestamp=0, x=15., y=10., xsize=5., ysize=8.)]

For o3:
cfr
o3 = [Object_State(name="o3", timestamp=0, x=14., y=10., xsize=5., ysize=8.)]

cbl
o3 = [Object_State(name="o3", timestamp=0, x=16., y=10., xsize=5., ysize=8.)]

clf
o3 = [Object_State(name="o3", timestamp=0, x=15., y=11., xsize=5., ysize=8.)]

crb
o3 = [Object_State(name="o3", timestamp=0, x=15., y=9., xsize=5., ysize=8.)]

Does these sound right? And when am I supposed to get cs* relations?
If they are not correct, maybe try to check the values before re-pushing? :P

This is right, ignoring the "s" relations which I don't bother with, see TODO on line https://github.com/cburbridge/strands_qsr_lib/blob/master/qsr_lib/src/qsrlib_qsrs/qsr_tpcc.py#L32. These "same" relations are very unlikely to occur in real floating point data...but ok I'll add them before this can be merged!

I see. Thanks. That's why I got confused. Happy to merge as it is and make an issue about it. Up to you whether you want to deal with it now or later.

Putting it in progress, let me know what you want to do per message ^^.

I have added the s relation. It won't ever come up in real data so it might be nicer to have some sort wider band for that relation, but it's not important now. Please merge this and if there are any further problems people can open issues for me to fix.
As a side note, I had a quick look at your unittests, but could not quickly see how I'm supposed to add one in your style. Can I just make a text file with the input and another with the expected QSRs?

Thanks!
Please rebase/resolve conflicts (would be faster than me doing it and making a PR to your branch), have a feeling it's only the qsrs/init.py.
Don't worry about the unnitest, I can easily do it as I need to update the multiple one also. Waiting for this PR to be merged before doing so though.

done.

I have added the s relation. It won't ever come up in real data so it might be nicer to have some sort wider band for that relation, but it's not important now. Please merge this and if there are any further problems people can open issues for me to fix.

You can use a threshold "q_factor" for that, used in others. Name it as quantisation_factor when you do and blame @cdondrup  for using british spelling in programming :P


Add storage_account_tls_version query for Ansible

This query evaluates if the latest version of TLS encryption (i.e. TLS1_2) is being used.
Closes #1636

Scan submitted to Checkmarx

Checkmarx SAST - Scan Summary & Details
Cx-SAST Summary
Total of 4 vulnerabilities 0 High 3 Medium 1 Low 0 Info
Violation Summary
 3 Medium
View more details on Checkmarx UI
Cx-SAST Details

Lines
Severity
Category
File
Link

30
Medium
Path_Traversal
cmd/builder/main.go
Checkmarx

52
Medium
Denial_Of_Service_Resource_Exhaustion
pkg/kics/service.go
Checkmarx

83
Medium
Denial_Of_Service_Resource_Exhaustion
test/main_test.go
Checkmarx


DOC: Scipy Documentation Pages Completely Down

Issue with current documentation:
The documentation page is dead, cannot access to any documentation of scipy.
Idea or request for content:
No response
Additional context (e.g. screenshots, GIFs)

Can you please remove the last dot from docs.scipy.org. and try again

No. I didn't input extra full-stop. For example I searched from Google or Bing the for 'scipy genhyperbolic', and get redirected through these links for example:
https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.genhyperbolic.html&ved=2ahUKEwjy0Njm15iGAxWdATQIHXfeADYQFnoECAcQAQ&usg=AOvVaw0PMNl4nkLNa_NBfWsb23gm

https://www.bing.com/ck/a?!&&p=ba903d6d99067eb6JmltdHM9MTcxNTk5MDQwMCZpZ3VpZD0wMTVkYjc0ZC1hMWU0LTZiYWItMDI2OS1hM2M5YTBiYjZhODYmaW5zaWQ9NTIwMw&ptn=3&ver=2&hsh=3&fclid=015db74d-a1e4-6bab-0269-a3c9a0bb6a86&u=a1aHR0cHM6Ly9kb2NzLnNjaXB5Lm9yZy9kb2Mvc2NpcHkvcmVmZXJlbmNlL2dlbmVyYXRlZC9zY2lweS5zdGF0cy5nZW5oeXBlcmJvbGljLmh0bWw&ntb=1

And then it gives me this:

Both of your search links are working for me. Despite what the browser is saying (to contact the website owners), this is typically a client-side problem, often to do with proxies or antivirus software.


Retain cfg-attributes on resources

When rust 1.43 lands as stable this will resolve #301 and allow for the kind of conditional compilation exemplified in the issue.
Tested on beta and nightly.

Lets wait for #307 to merge, then we can merge here.

Before merging I'd like to rebase and append the example to ci/script.sh

bors r+

bors r+


Multiple dates are not selected by default in edit form

Dear @adrienpoly,
I am using this plugin in my new project, I have a problem with multiple dates highlight in the edit form. The defaultDate data attribute is not working. Could you please help me with this.
Here am attaching the screenshot.

Regards,
Varma

hello @varmad
I had a quick look into it, defaultDate is a string in flatpickr. So maybe you could try to pass a string with the multiple dates rather than an array?
Adrien

Hello @adrienpoly
Yes, now it's working. Thank you so much for your quick reply.
I have noticed one more thing while selecting multiple dates from the date picker, by default comma and whitespace(", ") are added between the dates, so in edit form also you should populate the dates with the same format.
Example:
Create form:
2019-06-02, 2019-06-03, 2019-06-04(format from the calendar)
Edit form:
2019-06-02,2019-06-03,2019-06-04 (this format won't work)
2019-06-02, 2019-06-03, 2019-06-04 (this will work)
Varma

ok thanks for reporting it. I do plan to update this plugin shortly will look into this at that time


Update sbt-docker to 1.8.3

Updates se.marcuslonnberg:sbt-docker from 1.5.0 to 1.8.3.
GitHub Release Notes - Changelog - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "se.marcuslonnberg", artifactId = "sbt-docker" } ]

labels: sbt-plugin-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #805.


Allow stdin reading too: use - as filename

Dash is commonly used to represent stdin.
Can be handy, for example to compare files that only exist in git with actual files.

Thanks!


title get to "null" if you press off button first

I take your script to make a tab-notification.
Thank you very much for your work!!
But ... by pressing your off button first, your title will run into a "null" title,
This will appear if you take a onfocus element to switch off the tab-notification and get focus of the webside before a notification event is happen.
Workaorund is with a onload  function to activate and deactivate the notification function.

Hi @HoerMirAuf thanks for raising this issue.
I was able to reproduce the bug and have created a fix for it (#20) which has been published as part of v2.4.1.
You can demo this in use here:
https://flashing-page-title-notification.netlify.app/
Please let me know if you continue to have issues.


Domains and Hipache

I'm a little confused with setting up subdomains for Docker containers through Shipyard. Right now, the hack is to set up a Hipache container, and manually add keys to its Redis datastore, to assign a subdomain to a container's exposed port. Is that something Shipyard can do, without me messing with the Hipache instance?

You may be interested in https://github.com/ehazlett/interlock
On Thu, Apr 7, 2016 at 12:12 AM, Rudraksh MK<EMAIL_ADDRESS>wrote:

I'm a little confused with setting up subdomains for Docker containers
through Shipyard. Right now, the hack is to set up a Hipache container, and
manually add keys to its Redis datastore, to assign a subdomain to a
container's exposed port. Is that something Shipyard can do, without me
messing with the Hipache instance?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/shipyard/shipyard/issues/792

Thanks a bunch! Sure looks awesome, Interlock does.


Use syscall.Stdin for input handle. Fixes #1153

Summary
Cosign currently passes in the int '0' for the Stdin handle when reading from term. This unfortunately doesn't work on Windows, and should actually use syscall.Stdin to get the platform specific Stdin handle.
Ticket Link
Fixes #1153
Release Note

Bugfix to handle input on Windows platforms without crashing.

Thanks for the fix!

Codecov Report

Merging #1657 (bda89d0) into main (bdef009) will increase coverage by 0.68%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #1657      +/-   ##
==========================================
+ Coverage   28.08%   28.77%   +0.68%
==========================================
  Files         139      139
  Lines        8025     8139     +114
==========================================
+ Hits         2254     2342      +88
- Misses       5523     5544      +21
- Partials      248      253       +5

Impacted Files
Coverage Δ

pkg/cosign/common.go
0.00% <0.00%> (ø)

pkg/cosign/kubernetes/webhook/validator.go
80.49% <0.00%> (-1.22%)
:arrow_down:

pkg/cosign/tuf/client.go
62.34% <0.00%> (-0.95%)
:arrow_down:

pkg/apis/config/image_policies.go
69.38% <0.00%> (ø)

pkg/cosign/kubernetes/webhook/validation.go
80.21% <0.00%> (+1.64%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bdef009...bda89d0. Read the comment docs.

Sorry about spamming the CI/PR, let me figure out this lint issue on my own fork and I'll push when it's ready (it has to do with unconvert detecting an uneccessary conversion of 'int' on *nix but it's a uintptr on windows)

This should be working now and rebased onto the latest main branch. I "nolint'd" the int conversion, but the alternative is to build a platform specific ReadPassword to handle the int vs uintptr divergence. Happy to quickly build that out if you think it's the right direction.


Basic functions

Before creating forecast texts the classes and modules for the basic functions need to be created. The basic functions should be:

entry script to start the forecast
parameter handling fpr the calling parameter
loading the correct meteorological data
calculating the output values for a given forecast

Basic forecast text can be created with a given input file.


WIP: implement roi_align_2d

Sorry for the conflict with https://github.com/pfnet-research/chainer-compiler/commit/11c7ca1578be5bcd5adcaef840d6d7cb04f8f15b

Could you try removing this three lines?: https://github.com/pfnet-research/chainer-compiler/blob/master/scripts/ch2o_tests.py#L115

I removed the three lines.


filtering downloads using --exclusions option does not work with --build option

Describe the bug
I tried to download maven build artifacts with jfrog cli 2.13.0 using the --build option and filtering pom and jar files using --exclusions="*pom;*jar", but this seems not to work because it still downloads all the files including the pom and jar files.
When using path instead of build option the filtering works as expected.
Current behavior
.
Reproduction steps
No response
Expected behavior
No response
JFrog CLI version
2.13.0
Operating system type and version
macOS
JFrog Artifactory version
No response
JFrog Xray version
No response

Also in 2.32.0


[stable/prometheus-operator] Added checks for prometheusOperator being enabled when installing admission webhooks

Is this a new chart
No
What this PR does / why we need it:
Installs admission webhooks only if prometheusOperator is enabled.
Which issue this PR fixes

fixes #23326

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

cc @vsliouniaev @bismarck @gianrubio

/assign @vsliouniaev

/hold
Let’s hold PRs to stable/prometheus-operator chart for a short time until https://github.com/prometheus-community/community/issues/28#issuecomment-672933972 is resolved. If you want to help with that effort, to test etc, that would be great!

🚧 This chart is now deprecated.
Further development has moved to prometheus-community/helm-charts. The chart has been renamed kube-prometheus-stack to more clearly reflect that it installs the kube-prometheus project stack, within which Prometheus Operator is only one component.
If applicable, please re-open this PR there. Thanks!


[WIP] Expo build github action

This should be the code necessary for setting up an EAS build on push, tweaked to look similar to the other github actions.
It caches automatically with @actions/cache under the hood.

LGTM ship it!


Support padding CanvasTexture in TileSetAtlasSource

Now for a TileSetAtlasSource with use_texture_padding enabled and with a CanvasTexture assigned as texture the generated padded texture will also be a CanvasTexture, with the diffuse/normal/specular textures padded separately.
Fixes #61832 (https://github.com/godotengine/godot/issues/61832#issuecomment-1296140322).

Superseded by #83489.


Version 3.8.0 - Fatal error: Allowed memory size of 134217728 bytes exhausted

Bug report

OS: Windows/10
PHP CS Fixer 3.8.0 BerSzcz against war! by Fabien Potencier and Dariusz Ruminski.
PHP runtime: 8.1.4 (64bits version, having no php.ini)
php-cs-fixer fix . -vvv >output.txt 2>&1

Aborts with:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4194312 bytes) in C:\Users\Mirco\AppData\Roaming\Composer\vendor\friendsofphp\php-cs-fixer\src\Fixer\ClassNotation\OrderedClassElementsFixer.php on line 532

See also output.txt.
On having no php.ini
Notice: I don't have a php.ini when running php-cs-fixer.
I attached the output of "php -i" (the php used by php-cs-fixer.bat) in phpversion.txt. By default the PHP memory limit is 128M.
The project I'm working on is a PHP 5.6.40 project. So when running php.exe the 5.6.40 is used because that is in the path. 5.6.40 is not compatible with php-cs-fixer 3.8.0. To get around this limitation I adjusted the %appdata%\Composer\vendor\bin (global Composer) file php-cs-fixer.bat to:
@ECHO OFF
setlocal DISABLEDELAYEDEXPANSION
SET BIN_TARGET=%~dp0/php-cs-fixer
SET COMPOSER_RUNTIME_BIN_DIR=%~dp0
set PHPRC=
d:\projects\bin\php-8.1.4-x64\php.exe "%BIN_TARGET%" %*

Expectation
I would expect php-cs-fixer to set memory limit to unlimited. Because it is a CLI application. I see no point in aborting on a 64bits developer machine which has 16 GB of physical memory and around 10 GB free.
Possible solution
Adjust https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/aa805a8a4d8384c71a1367fceb57a081a4d969e4/php-cs-fixer#L15 with:
ini_set("memory_limit", -1);

@keradus

I don't know what you mean with "SCA tool"?
I don't know what you mean with "mess up with PHP settings of local env to for execution"?
If this refers to my proposed "ini_set()" statement, this statement is only an in process memory setting, not persisted on disk, not persisting to other scripts. Not messing up anything, only allowing php-cs-fixer to use as much memory as it needs.

ad 1 - Static Code Analysis tools, eg phpstan, psalm, phpcs. or even other, like composer, PHPUnit, etc.
ad 2 - i know it's not persistent, but why should the tool change it on their own? imagine you explicitly wanted to limit the memory to 2gb, and then tool escalate the limit to 10 gb on their own.

@keradus
You do have a point if you explicitly want to limit the memory usage of php-cs-fixer, my proposal would not allow that any more.
I have come up with an alternative solution. Set unlimited memory in php_cs.dist.php

@keradus
You do have a point if you explicitly want to limit the memory usage of php-cs-fixer, my proposal would not allow that any more.
I have come up with an alternative solution. Set unlimited memory in .php_cs.dist.php. This alternative solution works, so I'm closing this issue. I manually monitored the php memory usage and the peak was around 230 MB. Php-cs-fixer reported "Fixed all files in 20136.852 seconds, 182.000 MB memory used".
The odd thing is that I have to explicitly set the config file on the commandline. I expected it to be used automatically. But maybe that's just my lack of knowledge.
cd /d d:\projects\webpage\the-project
rem this directory contains .php_cs.dist.php file

php-cs-fixer fix . --config d:\projects\webpage\the-project\.php_cs.dist.php

<?php
// .php_cs.dist.php file

ini_set("memory_limit", -1);

$finder = PhpCsFixer\Finder::create()
    ->in(__DIR__);

$config = new PhpCsFixer\Config();
$config->setUsingCache(false)
    ->setRiskyAllowed(false)
    ->setRules([
        '@PSR2' => true,
        '@Symfony' => true,
    ])
    ->setFinder($finder)
;

return $config;

php-cs-fixer fix -vvv should pick up the config and show used config on verbose output
you mean you run it for 6h? sth really going wrong there. how big is your codebase? is it public, by any chance?

ad 1)
you use config file-name for PHP CS Fixer v2, but using v3 of the tool, rename config file to .php-cs-fixer.dist.php or point the filename as argument for fix command
ad 2)
ok, 250k files? so its slow, no further questions ;)

ad 1)
Thanx! After creating the .php-cs-fixer.dist.php file it is picked up automatically.
I saw a message from php-cs-fixer when I first ran it for version 3.8 complaining about .php_cs.dist file was an old file. I did not read the message correctly, I think, and just added the .php suffix resulting in .php_cs.dist.php. As the message was gone I thought everything was correct, not realizing that the filename and config-syntax has changed between v2 and v3.
It all works now. Thank you for your help!


PasswordCredentials expecting ClientInterface

Hi, I'm trying to run your plugin but an error stops me. Please help ;)
My code is:
$url = 'http://httpbin.org/get';
$oauth2Client = new Client(['base_url' => $url]);
$config = array(
'username' => 'myusername',
'password' => 'mypassword',
'client_id' => 'myclient',
'client_secret' => 'mysecret',
'scope' => 'administration', // Optional.
);
$grantType = new PasswordCredentials($oauth2Client, $config);
ERROR:
Catchable Fatal Error: Argument 1 passed to CommerceGuys\Guzzle\Plugin\Oauth2\GrantType\PasswordCredentials::__construct() must be an instance of Guzzle\Http\ClientInterface, instance of GuzzleHttp\Client given
composer.json requires:
"guzzlehttp/guzzle": "*",
"guzzlehttp/oauth-subscriber": "dev-master",
"commerceguys/guzzle-oauth2-plugin": "dev-master"

This was a Guzzle4/5 incompatibility (until recently the plugin only worked with Guzzle 3).


Issue 73 Fix Test Job Failure with CD Pipeline

I have added the db.session.close() in the test_model.py and test_service.py as suggested by the professor to see if this will solve the problem. If it doesn't solve the problem he also suggested that we could make a separate ElephantSQL instance just for testing

IBM Cloud toolchain: Delivery Pipeline deployed products to dev, including commit 95887df4df912af3fc1a0d222aff58c1de6ff099

IBM Cloud toolchain: Delivery Pipeline deployed products to prod, including commit 95887df4df912af3fc1a0d222aff58c1de6ff099


Question on Package Information

May you share the version information of each package used in the requirements file? Thank you!

Because of the last week's release of openai v1.x package, unfortunately it broke compatibility with the earlier versions of langchain packages. Just updated requirements.txt with a preset versions, although it's probably not a Best Practice 🙂.

These are the working versions. I'll remove their explicit references from the requirements.txt now, as it may introduce potential security issues if any vulnerabilities found in those versions in the future.
pymilvus==2.3.2
pypdf==3.17.0
langchain==0.0.253
openai==0.28.1
tiktoken==0.3.3

Thank you, @LazaUK!


MM-29869 Replace radio button Material with Compass icons

Summary
Replaces the Material  icons used for radio buttons (ex. setting the email settings) with the Compass icons.
Ticket Link
https://mattermost.atlassian.net/browse/MM-29869

Building app in separate branch.

Successfully building:
https://circleci.com/gh/mattermost/mattermost-mobile/11498
https://circleci.com/gh/mattermost/mattermost-mobile/11496

Artifact links:
https://11498-70265724-gh.circle-artifacts.com/0/tmp/artifacts/Mattermost_Beta.apk
https://11496-70265724-gh.circle-artifacts.com/0/tmp/artifacts/Mattermost_Beta.ipa

@matthewbirtch any noticeable difference in the icons?

@matthewbirtch any noticeable difference in the icons?

Nope, they should be exactly the same as the Material Design Icons

Cherry pick is scheduled.


--version option flag should not require a COMMAND

When running bench, you must provide a COMMAND even when just checking the program version...it would be nice to disable the COMMAND requirement for that particular option flag.
$ bench --version
Missing: COMMAND

Usage: bench COMMAND ([-I|--ci CI] [-G|--no-gc] [-L|--time-limit SECS]
             [--resamples COUNT] [--regress RESP:PRED..] [--raw FILE]
             [-o|--output FILE] [--csv FILE] [--junit FILE]
             [-v|--verbosity LEVEL] [-t|--template FILE] [-m|--match MATCH]
             [NAME...] | [-n|--iters ITERS] [-m|--match MATCH] [NAME...] |
             [-l|--list] | [--version])

Note that this does work as expected when using the --help option flag, which does not currently have the COMMAND requirement.

Thanks!


Serious issue of inconsistent content by deploying apk

From @ryc16 on October 18, 2016 9:28
Tell us about the problem
I recently started to build a debug apk for easy of deploying various devices for testing (which also similar production deployment) but I encountered very strange behavior. Apk got deployed but content sometimes show updated and sometimes not. Also same apk deploy to 2 devices but one shows correctly but another devices shown incorrectly. This could be easy to tell looking at UI already. The problem happen randomly so very painful. I wish more people try this way and see if more people got the same problem.
Steps:
#1. build apk by "tns build android".
#2. copy the xxx_debug.apk from platforms\android\build\outputs\apk to my share place.
#3. I installed the apk to 2 devices. note: both devices had early version installed.
From app info, both devices got the same, in term of content, they look different see the attachment for 2 devices.
Device1: content not updated look like old UI.
hw.zip
Device2: content did updated show correctly
note2.zip
On device 1, Also try clean cache, clean data on the device, uninstall package from device package manager, then install apk from my share place. Still had same problem - correct app version but incorrect content.
Any idea? How can I further troubleshoot? How could I know what content I am running on?
I am a bit worry what happen if this is on production environment, Would user will face this problem? If so, user wouldn't know about it. So this is very critical issue.
Which platform(s) does your issue occur on?
Only tried on Android not IOS
Please provide the following version numbers that your issue occurs with:
>tns info
All NativeScript components versions information
┌──────────────────┬─────────────────┬────────────────┬───────────────┐
│ Component        │ Current version │ Latest version │ Information   │
│ nativescript     │ 2.3.0           │ 2.3.0          │ Up to date    │
│ tns-core-modules │ 2.3.0           │ 2.3.0          │ Up to date    │
│ tns-android      │ 2.3.0           │ 2.3.0          │ Up to date    │
│ tns-ios          │                 │ 2.3.0          │ Not installed │
└──────────────────┴─────────────────┴────────────────┴───────────────┘

Copied from original issue: NativeScript/NativeScript#2918

From @ryc16 on October 18, 2016 11:5
In order to get the content up to date, I have to plug my device to my computer and use tns command.
tns livesync android --watch
Afterwards, everything works as expect.
Again, I am worry what would be happend if app is in production and install from ie google play store. Would this happen?

Hi @ryc16,
I couldn't reproduce your problem when I repeated the steps you provided. I tried building the default application and installing it by hand on different emulator api levels: 17, 19, 21, 24. The resulting UI is the same. If you want us to help you, can you provide a small repo causing the problem and be more specific with the api versions you tried running the application on?

@Plamen5kov, this doesn't happen all the time. I just wonder how does the android deployment works. Where does the content running at? For example, running at /data/data//files or it actually running on different location? When apk got deployed, does the process delete all files (including cache files if exist)? Would it possible some files is being locked on the file system?

@ryc16 when adb -r install <apk_name>-debug.apk is called, the application is copied on the device

for api level 17 in /data/app/<app_name>.apk and
for api level 24 in /data/app/<app_name>-1/base.apk

and then unpacked inside the folder you mentioned
/data/data/<full_app_name>/

When the app is uninstalled all these directories are deleted, so I can think of no way for the same app to have different behavior. I can suggest using the -r option on the adb command that will remove the previous application with the same name before installing the new one, or uninstalling manually before installation.
Adb command is:
adb install -r <apk_name>.apk

@Plamen5kov, I could try when it happen again (I didn't uninstall from device app manager which I think the same so not sure if adb -r helps). Is adb -r is the regular deployment process or handle by tns android cli? If your suggestion is just for troubleshooting, then it is ok otherwise this wouldn't be production solution. So there is no cache mechanism of running content, right? What about the javascript layer? Hope other people could replicate this problem.

Is adb -r is the regular deployment process or handle by tns android cli?

@ryc16, yes the CLI uses -r as the default command when installing the application as you can see here.

So there is no cache mechanism of running content, right?

I can't tell you if android has a cache mechanism, I haven't looked into it.

If your suggestion is just for troubleshooting, then it is ok otherwise this wouldn't be production solution.

I can't reproduce the issues at hand, so it's a suggestion to help you isolate the problem. If we manage to do that, we can figure it out.

I've seen this issue too... very strange!
I updated an XML file to fix a spelling issue, tested it on a real Nexus 7 device worked fine. Uploaded the debug apk to a real Motorola G and the old UI was still there!

Now more people are able to catch this strange problem. This kind of problem is very scary if it is on production. We don't know if user really running/showing the latest content.

Hi @ryc16,
I agree with you, but you have to understand, if we can't reproduce your problem, we can't fix it. I would be grateful if you find a scenario, where this bug can be reproduced, so we can take it on.

Hi @Plamen5kov, understood. it is not consistent to me so not sure how to let you reproduce the problem.
@therevills, is it consistent to you? would you have anything for @Plamen5kov .

@ryc16 did you mange to isolate the issue, or find a way for us to reproduce it?

I'm closing this issue, because there's no response, but feel free to reopen it again, if you find a way to reproduce it.

Hey, I am also facing the same issue. It's very frustrating. In my scenario, I added few images on app resources and when I tried to run the app. It shows me the previous content.

I am using NativeSctipt 2.5

I got the issue. The issue was I changed the NativeSctipt id but had not changed in app.gradle therefore it was giving me this kind issue. For more information you can refer https://GitHub.com/NativeSctipt/nativescript-cli/issues/2092

Hi @mayureshjadhav,
Could you provide steps to reproduce the problem. Keep in mind that when using livesync all changes made are passed directly to the device, and when all fixes using livesync are done a normal build needs to be made so a new .apk is generated: tns build android/ios.

Will close this issue because there's no steps to reproduce and lack of interest. Feel free to reopen if someone can provide steps to reproduce.

@Plamen5kov please re-open...  Believe it or not; I just saw this issue.  Wasted about 2 hours of my time.  ;-(
However, I know what happened, not 100% if it is easily duplicatable, I didn't retry -- but I know how I fixed it and what caused it...

Using TNS 3.0 CLI  (probably doesn't matter as it isn't a CLI issue afaik).
I was tonight doing a lot of livesyn'ing while building an app.
When I went to go test a "build" version (i.e. I did a tns build android).
De-installed app from device.
Then I copied the file over to the emulator (geny you can drag & drop).     Then when I ran it; it was automatically pulling the "sync" from the /data/local/tmp thinking it apparently needed to "update"...  Of course the version in that folder was outdated as I had two new fixes...

Blew my mind everytime I started the app and it did the old behavior.  Spend a while debugging again until I realized my source didn't have the two fixes.   Ugh!!!
Open the debugger and it showed the old JS, open up the APK and it had the new files.  Finally looked on the raw device, and it had the old files...
Once I deleted the /data/local/tmp directory for my app; then the install worked great...
So somehow the LiveSync on startup was getting confused and thought the /data/local/tmp was a later version even though a bit of time went by while I was building new version (no more livesync'g as I wanted a test build to give my clients)...

@NathanaelA thanks for describing the process in details.
It just came to my attention that we face the problem internally too. I can finally confirm that this is indeed an issue with temp directory not being erased on certain devices, despite permissions being granted.

Addressed with changes in PR https://github.com/NativeScript/android-runtime/pull/743 as well as some modifications inside the CLI version 3.0.0

@Pip3r4o how do we go about getting this update?
Since I couldn't quite figure it out, this was a temporary workaround for me.
Based off of the comments, I was able to delete the files in /data/local/tmp/ using adb and now my apk that I installed is actually using the files from the apk.
to get the directory name, you can find it by running this
adb shell ls /data/local/tmp/

Then, I did
adb shell rm -rf /data/local/tmp/org.nativescript.MYAPPNAME

@cjdreiss 3.0.0 will be officially released next week. Using the 3.0 CLI and installing the 3.0 android runtime in your projects (even if you still use 2.5 modules) should prevent your files from going stale.

+1 Reported via t.1119637 with tns-android 3.2.0

@NickIliev the changes released earlier this year unfortunately do not fix the problem occurring on Samsung devices.

I can confirm this still happens with 3.3.1
to reproduce:

tns run android
kill livesync
make a noticeable edit to a template or something
either

tns build android and then adb install path/to/appName-debug.apk to the same device/emulator that the livesync was hooked up to

or tns deploy android to the same device/emulator that the livesync was hooked up to

the change/s will not be noticeable, and most appearance/behaviour will instead reflect the most recent livesync

This problem doesn't seem to affect release builds (tns build android --release) or clean deploy (tns deploy android --clean)

what is the solution plz


Upgrade Lucene Codec to Lucene99

Description
This PR includes the following changes:

Add new codec that wraps Lucene Codec 99 and make it default
Fix import statements for Lucene Codec 95
Change CorrelationCode990 and CorrelationCodec950 constructor to type PerFieldKnnVectorsFormat to abstract the field type.

Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

why do we need new codec? what was the previous code being used?

are there any backward compatibility issues when customer upgrades from a version with previous codec to this?

Asked @sbcd90 yesterday about keeping the old codec and he said it would be good to keep. K-nn keeps their old codecs for backwards compatibility which is why I kept both old and new codec. Don't know how to test this as we don't have a BWC workflow set up yet.


Move repo to Jazzband?

Hi @aljosa, @Natim,
What would be your opinion on moving this repository to Jazzband (jazzband.co/)? I don't mean to hurt you, but maintainership is a bit minimal currently, so it might help to revitalize this package.

Hello @claudep
Thank you for asking the question. I was also looking forward to add django-downloadview to JazzBand. I also think it might benefit the library to do so.
The problem is maintaining a library when one is not actively using it anymore.
Regarding jazzband, I think we need to improve the tests, it is currently too easy to break the lib. If you have any ideas I am interested.

@claudep yes, @Natim suggested it before but just didn't have time to move forward with the idea.
What do we need to do?

Here you can find instructions: https://jazzband.co/about/guidelines
Transferring the repo to Jazzband on Github seems the first step to do.

Regarding jazzband, I think we need to improve the tests, it is currently too easy to break the lib. If you have any ideas I am interested.

You mean JS tests? Django is using QUnit (https://qunitjs.com/) to test its JS code, maybe a hint…

Transferring the repo to Jazzband on Github seems the first step to do.

@aljosa if you agree let's do that :)

@Natim I've just asked on irc if it's ok to transfer the repo and work on issues/details like tests later

@Natim after reading more on the website it looks to me it's ok to transfer so I did

:heart_eyes:


New State: Karnataka (KA)

State Name: Karnataka
Bulletin link: Link
Defaults

[ ] Data extraction done | HowTo
[ ] Added entry into landing page | HowTo
[ ] Added DB Schema in the Wiki | Wiki

Hi @MayankAgarwal  I am interested to take up this exercise. Please let me know if you can assign it.

Hi @sridhatta: Welcome to the project and I'm glad that you want to contribute. I've assigned the issue to your name. Here's how you can get started:

If you haven't already, you can go through the procedure to add new states in this wiki.
Please join the Slack channel for this project (https://join.slack.com/t/covid-19-india-data/shared_invite/zt-uej5v98i-mjggggkLMASKFbZRXzq4x), so that we can collaborate better.
Fork this repo, and start adding code for data extraction from Karnataka.

KA bulletins require table extraction from images. We are planning on using Amazon Textract API to extract the same. We can discuss more on this matter (how to utilize this API, if you have other ideas, etc.) in the Slack channel.

Please let me know if you have any issues or questions.

Hi @MayankAgarwal  I am not able to join the slack channel as the link appears to be expired can u please resend it?


ILD reco tutorial differences between zsh and bash

Hello!
I'm trying to run the ILD reconstruction tutorial but I'm running into an error at the last step: simulation (using ddsim) works fine, but at the reconstruction step (using k4run ILDReconstruction.py) I see the multiple instances of the following messages.
[ ERROR "MyAdd4MomCovMatrixCharged"]  Warning: null charge particle! This case is not supported

and towards the end of the execution (after the event loop):
Failed to create calo hit: STATUS_CODE_NOT_FOUND
PandoraApi::CaloHit::Create(m_pandora, caloHitParameters) throw STATUS_CODE_NOT_FOUND
    in function: CreateMuonCaloHits
    in file:     /tmp/root/spack-stage/spack-stage-ddmarlinpandora-0.12.01-p742e5lzpihslq2zfjsto66iki2aaskn/spack-src/src/DDCaloHitCreator.cc line#: 388
[ ERROR "MyDDMarlinPandora"] Failed to extract muon hit: STATUS_CODE_NOT_FOUND
EDM4hepOutput       ERROR Could not write event, because the following collections are not present: V0RecoParticles, V0RecoParticles_particleIDs, V0Vertices
EDM4hepOutput       ERROR Maximum number of errors ( 'ErrorMax':1) reached.
EventLoopMgr      WARNING Execution of algorithm EDM4hepOutput failed
EventLoopMgr        ERROR Error processing event loop.
EventLoopMgr        ERROR Terminating event processing loop due to errors
EventLoopMgr        ERROR Terminating event processing loop due to errors

and finally the last line of the output
ApplicationMgr      ERROR Application Manager Terminated with error code 3

I noticed that the files are still created though. Now the actually surprising thing is, if I run the EXACT same setup using bash instead of zsh (I'm using zsh by default), I only see the following message:
PandoraApi::CaloHit::Create(m_pandora, caloHitParameters) throw STATUS_CODE_NOT_FOUND
    in function: CreateMuonCaloHits
    in file:     /tmp/root/spack-stage/spack-stage-ddmarlinpandora-0.12.01-p742e5lzpihslq2zfjsto66iki2aaskn/spack-src/src/DDCaloHitCreator.cc line#: 388

but not the Warning: null charge particle, nor the rest of the above EDM4hepOutput or EventLoopMgr errors/warnings. The application finally exits gracefully.
Comparing the root files (events tree), the bash version contains 3 events while the zsh version contains 2 events, and the branches also differ.
I'm a bit perplexed that the differences are only due to which shell is used... (I put the snippet below in a script, and run it with ./script.sh or bash script.sh, from a fresh terminal)
Antoine

Steps to reproduce: follow the ILD reco tutorial (Setup + Simu + Reco/Using ILDReconstruction.py).

OS version: Alma9 (DESY NAF)
key4hep stack: release 2024-04-12
compiler, ROOT ...: shipped with key4hep stack.

source /cvmfs/sw.hsf.org/key4hep/setup.sh -r 2024-04-12

mkdir key4hep_tut_ild_reco
cd key4hep_tut_ild_reco

git clone https://github.com/iLCSoft/ILDConfig
cd ILDConfig/StandardConfig/production
wget https://raw.githubusercontent.com/key4hep/key4hep-tutorials/main/gaudi_ild_reco/input_files/zh_mumu.slcio

ddsim --compactFile $k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml \
      --steeringFile ddsim_steer.py \
      --inputFiles zh_mumu.slcio \
      --outputFile zh_mumu_SIM.edm4hep.root

k4run ILDReconstruction.py \
      --detectorModel=ILD_l5_o1_v02 \
      --inputFiles=zh_mumu_SIM.edm4hep.root \
      --outputFileBase=zh_mumu

Hello again,
I'm closing this issue as results are not reproducible: running the same command multiple times in a fresh shell might or not yield the error.
Nevertheless: some precisions about the tests I conducted above.

I figured out what I thought were clean shells might not have been: some had the key4hep release already setup before running the script. This should not matter as the release is setup in the script anyways, but still.
As I didn't put a shabang at the beginning of the script, I assumed calling ./myscript.sh would use the current shell (zsh in my case) as interpreter. After checking, in the absence of shabang, sh is rather called by default, which on my system is softlinked to bash. This makes even less sense as calling ./myscript.sh and bash myscript.sh should then run the same interpreter (and hopefully give the same results)...
To add even more to my confusion, I redid systematic checks using fresh shells and explicitly calling the script with zsh, bash or sh, which yield inconsistent results.
I suspected environment variables or my zsh rc would be an issue so I tried the previous line adding env -i in front, and combining with passing the options zsh -f (to ignore rc files) and bash --norc --noprofile. Again the results are inconsistent.

TL;DR: I have no idea what cause this difference in the running of ILDReco.


Support *args and **kwargs in operation function signatures

Description
This is a follow up to #268 to allow decorating operation functions where the decorator changes the signature to generic *args, **kwargs.
Expected behaviour
Parameters should be mapped for operation functions with *args and/or **kwargs.
Actual behaviour
Parameters not directly matching the function signature are silently dropped (kwargs are not populated).
Steps to reproduce

Create a simple Swagger YAML with one operation having at least one query parameter "foo".
Define the operation function with only *args and **kwargs.
The Swagger query parameter "foo" is not passed in **kwargs.

Released in https://github.com/zalando/connexion/releases/tag/1.0.112


HttpException (HttpException: Failed to parse header value)

Hello! I want to write a mobile application for a keenetic router. But I get an HttpException (HttpException: Failed to parse header value) error when I make a request to the Keenetic router.
void httpp() async {
   HttpClient httpClient = new HttpClient();
   HttpClientRequest request =
       await httpClient.get('<IP_ADDRESS>', 80, '/auth');
   // it's polite to send the body length to the server

   request.headers.set('Accept-Encoding', '*', preserveHeaderCase: true);
   request.headers.set('Accept', '*/*', preserveHeaderCase: true);
   request.headers
       .set('Transfer-Encoding', 'chunked', preserveHeaderCase: true);
   request.headers.set('Connection', 'keep-alive', preserveHeaderCase: true);
   HttpClientResponse response = await request.close(); // HttpException (HttpException: Failed to parse header value)
   final chunks = await response.transform(utf8.decoder).toList();
   final bbb = chunks.join(',');
   httpClient.close();
   print(bbb);
   print(response);
 }

I verified via Wireshark that the request is being processed and I am getting a response headers like this:
{'Server': 'Web server', 'Date': 'Thu, 29 Sep 2022 22:21:56 GMT', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Set-Cookie': 'TMAKGAGUCQNB=HHMDXTJAKYRNHCNS; Path=/', 'WWW-Authenticate': 'x-ndw2-interactive realm="Keenetic Air" challenge="MQWUWIVNDDMINFPVMUOBNWFSIRBGPTKI" session_id="HHMDXTJAKYRNHCNS" session_cookie="TMAKGAGUCQNB"', 'X-NDM-Realm': 'Keenetic Air', 'X-NDM-Challenge': 'MQWUWIVNDDMINFPVMUOBNWFSIRBGPTKI'}
But he still couldn't solve the problem. I will be grateful for help.
[✓] Flutter (Channel stable, 3.3.6, on macOS 12.6 21G115 darwin-arm, locale ru-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.0)
[✗] Chrome - develop for the web (Cannot find Chrome executable at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome)
! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[✓] Android Studio (version 2021.3)
[✓] VS Code (version 1.73.0)
[✓] Connected device (1 available)
[✓] HTTP Host Availability
! Doctor found issues in 1 category.
Dart SDK version: 2.18.2 (stable) (Tue Sep 27 13:24:11 2022 +0200) on "macos_arm64"

The Set -Cookie header looks malformed. A space in the header name isn't allowed and will not work. It seems like a bug in your keenetic router? There's not so much you can do if it's not responding in valid HTTP.
It looks like you posted the HTTP response as decoded by wireshark into JSON format. Can you post the raw HTTP request and raw HTTP response? The exact characters sent and received will help a lot when we think about cases like this, where we're dealing with a server that does not implement the specification and barely work.

The Set -Cookie header looks malformed. A space in the header name isn't allowed and will not work. It seems like a bug in your keenetic router? There's not so much you can do if it's not responding in valid HTTP.
It looks like you posted the HTTP response as decoded by wireshark into JSON format. Can you post the raw HTTP request and raw HTTP response? The exact characters sent and received will help a lot when we think about cases like this, where we're dealing with a server that does not implement the specification and barely work.

I am sorry. I made a mistake when I copied the headers.  There is no space character.

Request:
GET /auth HTTP/1.1
user-agent: Dart/2.18 (dart:io)
Connection: keep-alive
Accept: /
Accept-Encoding: *
host: <IP_ADDRESS>
474554202f6175746820485454502f312e310d0a757365722d6167656e743a20446172742f322e31382028646172743a696f290d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a4163636570743a202a2f2a0d0a4163636570742d456e636f64696e673a202a0d0a686f73743a203139322e3136382e312e310d0a0d0a
Response:
HTTP/1.1 401 Unauthorized
Server: Web server
Date: Thu, 29 Sep 2022 21:55:34 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: LTLBXITXDJFHLUF=OCKUWIXZJPYLQFYU; Path=/
WWW-Authenticate: x-ndw2-interactive realm="Keenetic Air" challenge="KVAZHEAUGGWRZODBBJSBFWMYRXAYVVPN" session_id="OCKUWIXZJPYLQFYU" session_cookie="LTLBXITXDJFHLUF"
X-NDM-Realm: Keenetic Air
X-NDM-Challenge: KVAZHEAUGGWRZODBBJSBFWMYRXAYVVPN
485454502f312e312034303120556e617574686f72697a65640d0a5365727665723a20576562207365727665720d0a446174653a205468752c2032392053657020323032322032313a35353a333420474d540d0a5472616e736665722d456e636f64696e673a206368756e6b65640d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a5365742d436f6f6b69653a204c544c4258495458444a46484c55463d4f434b555749585a4a50594c514659553b20506174683d2f0d0a5757572d41757468656e7469636174653a20782d6e6477322d696e746572616374697665207265616c6d3d224b65656e657469632041697222206368616c6c656e67653d224b56415a48454155474757525a4f4442424a534246574d59525841595656504e222073657373696f6e5f69643d224f434b555749585a4a50594c51465955222073657373696f6e5f636f6f6b69653d224c544c4258495458444a46484c5546220d0a582d4e444d2d5265616c6d3a204b65656e65746963204169720d0a582d4e444d2d4368616c6c656e67653a204b56415a48454155474757525a4f4442424a534246574d59525841595656504e0d0a0d0a

Where did you get the response data from? Is this an print from Dart or from curl or something?

Where did you get the response data from? Is this an print from Dart or from curl or something?

I am getting this exception in Dart. I solved this problem with Flutter platform channels.

Sorry, I meant where did the output that you posted i.e.
HTTP/1.1 401 Unauthorized
Server: Web server
Date: Thu, 29 Sep 2022 21:55:34 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: LTLBXITXDJFHLUF=OCKUWIXZJPYLQFYU; Path=/
WWW-Authenticate: x-ndw2-interactive realm="Keenetic Air" challenge="KVAZHEAUGGWRZODBBJSBFWMYRXAYVVPN" session_id="OCKUWIXZJPYLQFYU" session_cookie="LTLBXITXDJFHLUF"
X-NDM-Realm: Keenetic Air
X-NDM-Challenge: KVAZHEAUGGWRZODBBJSBFWMYRXAYVVPN

485454502f312e312034303120556e617574686f72697a65640d0a5365727665723a20576562207365727665720d0a446174653a205468752c2032392053657020323032322032313a35353a333420474d540d0a5472616e736665722d456e636f64696e673a206368756e6b65640d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a5365742d436f6f6b69653a204c544c4258495458444a46484c55463d4f434b555749585a4a50594c514659553b20506174683d2f0d0a5757572d41757468656e7469636174653a20782d6e6477322d696e746572616374697665207265616c6d3d224b65656e657469632041697222206368616c6c656e67653d224b56415a48454155474757525a4f4442424a534246574d59525841595656504e222073657373696f6e5f69643d224f434b555749585a4a50594c51465955222073657373696f6e5f636f6f6b69653d224c544c4258495458444a46484c5546220d0a582d4e444d2d5265616c6d3a204b65656e65746963204169720d0a582d4e444d2d4368616c6c656e67653a204b56415a48454155474757525a4f4442424a534246574d59525841595656504e0d0a0d0a

come frome?


Add Authorization methods

Allow authorization methods like Basic.
CLI usage is not defined yet.

Do you think it would be a good idea to store the auth method separately from the handle, so we could re-use it across different handles?

I think it does too. Now that Environments are a thing it would be much easier to implement that way. Just like the cookie engine works nowadays.
Just as cookies, it shouldn't be tracked either. But if we keep adding untracked files, it would be better to think of a 2.x to make it a pattern to have a ./secrets whenever it is needed for untracked files. Gitignore would be:
.quartz/**/secrets


compare with WDSR and EDSR

Hi, thanks for the code. I notice that your model is developed by WDSR, would you please provide the comparision results with WDSR and EDSR? Thanks.
https://github.com/JiahuiYu/wdsr_ntire2018/blob/master/wdsr_b.py

@zhengsx  hello, in the paper, we add a comparison of our model and WDSR baseline. EDSR is a model with a large amount of parameters. So we didn't compare it to our model. But you can compare the baseline of WDSR and EDSR on the x2 scale on the DIV2K val dataset.

Network
Parameters
DIV2K (val) PSNR

EDSR Baseline
1,372,318
34.61

WDSR Baseline
1,190,100
34.77

AWSRN
1,396,872
34.81

I'm very appreciate that.


Inverter CEC upgrade script trips on existing inverters

Describe the bug
The version upgrade script is supposed to check to see if an inverter has been removed from the CEC library and make the user choose a new inverter if their project's inverter is no longer in the library. However, this appears to be happening for all inverters.
To Reproduce
Steps to reproduce the behavior:

Open the attached SAM file in 2023.12.01 (or develop)
inverter_upgrade_test.zip
The version upgrade script forces choosing a new inverter when the inverter exists in the library.

Expected behavior
The version upgrade script should skip the inverter replacement for this file.
Screenshots

Operating System and Version

OS: Windows
Version 2023.12.01/develop

I'll take a look at this.

Still looking into this. The right hand side of the  '?@' command on line 135 of version.lk has a m_type of 36, while the right hand side has m_type of 35.


Upgrade to OCP 4.5.2

Manual installation worked

http://<IP_ADDRESS>:8080/job/test-infra/1820/console
passed


Mark as invalid with message from server?

Is there an easy way to mark an ampersand-input-view as invalid with a serverside error message?
e.g. i have an registration-view with email and password fields and get serverside validation information for one or both.

I wrote an async form validation example gist awhile back, it may be what you're looking for:
https://gist.github.com/wraithgar/7fc72e5fb72502c99dcf
You would of course want to do an ajax call in lieu of the setTImeout call but hopefully this gets you started at least.


Upgrade to Jest 23 for JS unit tests

This upgrades the Jest version used in Wagtail. @helenb and I were having some issues updating snapshot tests because of a bug in Jest, that has been fixed in v23.0.0.
Jest 24 is out, but upgrading to that would also require we upgrade Babel from v6 to v7 – which is quite a bit of work.
After the upgrade, I also had to change how some of the Draftail initialisation tests were written. I couldn’t pinpoint which change caused that but I wouldn’t be surprised if jsdom had changed the behavior of how they parse script tags in HTML that caused those tests to break.

I have tested this on my local build. The steps taken are as follows:

Check out a branch with @thibaudcolas 's changes
Run npm install in /vagrant/wagtail on the VM
Make a markup change to wagtail/client/src/components/Explorer/ExplorerPanel.js which will mean it no longer matches the previous snapshot
Commit the change
Run npm run build in /vagrant/wagtail on the VM
Run npm run test:unit:watch in /vagrant/wagtail on the VM
When failures are reported, select u to update the snapshots
Failed snapshots are successfully updated.

Possible issues:

At one point I attempted to re-run only failed tests after running all the tests (option f), and it claimed that there were no failed tests.
The tests were a lot slower to run than the previous version of jest which ran all the tests in about 30 seconds (sample output while running the tests is below)

PASS  client/src/components/Draftail/sources/ModalWorkflowSource.test.js (9.498s)
 PASS  client/src/components/Draftail/decorators/TooltipEntity.test.js (8.292s)
 PASS  client/src/components/Draftail/blocks/ImageBlock.test.js (8.311s)
 PASS  client/src/components/Draftail/decorators/Link.test.js (8.213s)
 PASS  client/src/components/Draftail/DraftUtils.test.js (7.776s)
 PASS  client/src/components/Draftail/decorators/Document.test.js (8.177s)
 PASS  client/src/components/Draftail/blocks/MediaBlock.test.js (8.067s)
 PASS  client/src/components/Draftail/index.test.js (9.382s)
 PASS  wagtail/admin/static_src/wagtailadmin/app/draftail.entry.test.js (8.76s)
 PASS  client/src/components/Explorer/Explorer.test.js (8.071s)
 PASS  client/src/components/Draftail/EditorFallback/EditorFallback.test.js (6.974s)
 PASS  wagtail/admin/static_src/wagtailadmin/app/wagtailadmin.entry.test.js (7.521s)
 PASS  client/src/index.test.js (7.487s)
 PASS  client/src/components/Draftail/blocks/EmbedBlock.test.js (7.983s)
 PASS  client/src/components/Portal/Portal.test.js (6.33s)
 PASS  client/src/components/Explorer/reducers/explorer.test.js (5.68s)
 PASS  wagtail/admin/static_src/wagtailadmin/js/page-editor.test.js (7.966s)
 PASS  client/src/components/Explorer/ExplorerItem.test.js (6.487s)
 PASS  client/src/components/Transition/Transition.test.js (6.455s)

 RUNS  client/src/api/client.test.js

Test Suites: 19 passed, 19 of 36 total
Tests:       124 passed, 124 total
Snapshots:   47 passed, 47 total
Time:        284s

Thanks @helenb! The build time is looking very similar in CircleCI between this PR and others, so I think your test time might have been due to your laptop’s CPU load at the time? Or comparing time running the tests in Vagrant vs on your host machine?

Thanks @helenb  for your great validation instructions.
Merged in via 471823f2384f7c8d64d7866025f95c8eb929056f


Arbitrage Page

Add frontend page to display profitability paths

[x] Add drop down menus for start and end currencies
[x] Add drop down menu for currency exclusions
[x] Add drop down for limiting edges
[x] Add api calls for arbitrageData
[x] Display optimal rate conversion upon button press
[x] Display optimal paths upon button press

Closed as of 4646cf6860f966b3391f8bb0faef88a8b3faead2


Descheduler acting on nodes without label in nodeSelector

We are using karpenter to provision capacity as needed and the descheduler to evict pods and tear down those nodes based on a HighNodeUtilization strategy.  We are also using nodeSelector so that the descheduler acts only on a subset of our nodes.  Our policy looks like
nodeSelector:
  karpenter.sh/provisioner-name: karpenter-provisioner
deschedulerPolicy:
  strategies:
    HighNodeUtilization:
      enabled: true
      params:
        nodeFit: true
        nodeResourceUtilizationThresholds:
          thresholds:
            cpu: 30
            memory: 60
            pods: 75

However, the descheduler logs show it is trying to act on nodes that do not have this label
I0223 19:40:57.676274       1 highnodeutilization.go:91] "Number of underutilized nodes" totalNumber=2
I0223 19:40:57.676297       1 nodeutilization.go:223] "Total capacity to be moved" CPU=4500 Mem=34888605696 Pods=92
I0223 19:40:57.676310       1 nodeutilization.go:226] "Evicting pods from node" node="ip-10-3-141-88.us-west-2.compute.internal" usage=map[cpu:125m memory:0 pods:3]
I0223 19:40:57.676340       1 node.go:123] "Pod can possibly be scheduled on a different node" pod="gitlab-runner/runner-voaz8b2r-project-32000304-concurrent-1qrd6w" node="ip-10-3-132-203.us-west-2.compute.internal"
I0223 19:40:57.676399       1 nodeutilization.go:229] "Pods on node" node="ip-10-3-141-88.us-west-2.compute.internal" allPods=3 nonRemovablePods=3 removablePods=0
I0223 19:40:57.676406       1 nodeutilization.go:232] "No removable pods on node, try next node" node="ip-10-3-141-88.us-west-2.compute.internal"

Our gitlab runners run on nodes created by karpenter but by a different provisioner and therefore do not have the label indicated in our policy.  I have confirmed this.
Additionally, when the descheduler builds a list of nodes as indicated in the logs with
I0224 15:18:51.243243       1 node.go:46] "Node lister returned empty list, now fetch directly"

all of our nodes are listed regardless of whether or not the nodeSelector is satisfied.  This may be expected behavior, but seems suspect.

Here is more information showing the descheduler acting on nodes without the nodeSelector label:
I0224 15:38:51.269213       1 nodeutilization.go:226] "Evicting pods from node" node="ip-10-3-138-169.us-west-2.compute.internal" usage=map[cpu:125m memory:0 pods:2]

and output from kubectl describe node:
$ kubectl describe node ip-10-3-138-169.us-west-2.compute.internal|grep 'karpenter.sh/provisioner-name'
                    karpenter.sh/provisioner-name=gitlab-runner-provisioner-karpenter-provisioner

Thanks for the report @bwmetcalf, that does seem weird because the list of nodes passed to each strategy should be based on the labelSelector as set in ReadyNodes (https://github.com/kubernetes-sigs/descheduler/blob/54c50c53902c580f3752d5068fbfc18a252b43da/pkg/descheduler/node/node.go#L48)
Though I know the utilization strategies in particular have some limits on the ability to restrict nodes/pods, we'll have to check if that comes into play here.
/kind bug

I noticed the documentation here https://github.com/kubernetes-sigs/descheduler#policy-and-strategies for nodeSelector shows an example as
nodeSelector: prod=dev

but it's a map that should be passed in to the chart which is consistent with https://github.com/kubernetes-sigs/descheduler/blob/3ab0268c5aabdb487e0f6ec4d924511f0b8e38f9/charts/descheduler/values.yaml#L76.
Just pointing this out just in case there is something there.

@bwmetcalf that's interesting, are you deploying with the helm chart? Could you please share you descheduler policy config?

Yes, we are deploying with the helm chart and using kind: Deployment.

Maybe this is a mis-match between the type as defined in the chart and what's parsed by the node selector helpers.
Do you have any way to try deploying it manually to use the label=value format in your descheduler policy, and see if that shows the same behavior? You might be able to even just modify the policy configmap and kill the Pod if descheduler is already deployed

Editing the deployment "live" results in
error: deployments.apps "descheduler" is invalid

when trying to use karpenter.sh/provisioner-name=karpenter-provisioner.  It seems the docs are incorrect and this is a red herring.

Could you please paste the steps and full object for trying to make that change? It's odd that updating the policy configmap gave an error with the deployment

Ah.  That seems to be the problem.  After correcting this, the behavior seems to be correct.  Sorry for the confusion.  I will close this issue.

No worries! Glad we could resolve it. We should update the documentation to explain that better, it is confusing with 2 of those


HAPROXY_0_BACKEND_HEAD newline character cause problem

I try to follow this way to increase my backend service's timeout: https://github.com/mesosphere/marathon-lb/wiki#setting-timeout-for-long-lived-socket-connections
I found haproxy config be generated properly from: :9090/_haproxy_getconfig which is good. But I found my marathon application failed to start up. It's because in the label HAPROXY_0_BACKEND_HEAD contains \n which cause mesos not able to launch docker application(mesos try to set this lable as a docker's env var). The error shows in mesos: docker: poorly formatted environment: variable 'balance {balance}' has white spaces.
So I try to avoid use \n in label HAPROXY_0_BACKEND_HEAD. But that cause haproxy config syntax error.
Any help?

Hi,
Where are you configuring this?  The labels from that example should be configured as labels on the target application, not on Marathon-LB.  Also, they should be configured as labels, not as environment variables.
Can you show me the marathon json definition you're trying to use?
Thanks,
Justin Lee

I am having the same issue.  I set the HAPROXY_0_BACKEND_HEAD in my app to:
{
  "labels": {
    "HAPROXY_0_BACKEND_HEAD": "backend {backend}\n  balance {balance}\n  mode {mode}\n  timeout server 30m\n  timeout client 30m\n"
  }
}

My app fails to start.  In my app's stderr, I see:
docker: poorly formatted environment: variable 'balance {balance}' has white spaces.

To the best of my understanding, Mesos tries to inject the environment variable MARATHON_APP_LABEL_HAPROXY_0_BACKEND_HEAD.  My guess is this is where it fails.
Mesos version: 1.2.0
Marathon version: 1.4.11

OK. I confirmed that this is because of the mesos version. I upgraded my mesos slave version to 1.3.2 and it solved the problem.


Getting CSV input on PY3 to work

Motivation for changes:
csv input was not working on PY3
Detailed changes:
CSV reader in Python 2 is broken and does not support unicode strings. It therefore needs byte strings, but Python 3 needs unicode strings.
csv plugin has been changed to transform the string to the proper format based on Python version.
Addressed issues:
http://discuss.flexget.com/t/input-csv-py-not-working-on-python3/2797

Thanks


Ssl: extract ConnectionInfo implementation from SslHandshake

Signed-off-by: Dan Zhang<EMAIL_ADDRESS>The new class Ssl::ConnectionInfoImpl is going to be shared with Quic connections.
Risk Level: low, refactoring only
Testing: existing tests

/wait

/assign @alyssawilk

/retest

/retest

may need a main merge due to CI changes sorry


Add STDIN support to tsz

Is it possible to add STDIN support to tsz ?
I would like to do the following:
dmesg | tsz  --stdin dmesg.txt

Currently you would have to write a temporary file first
dmesg > /tmp/dmesg.txt
tsz /tmp/dmesg.txt

tsz reads the data sent by the client from stdin.
Considering dmesg | tsz  --stdin dmesg.txt, the data sent by the client will be received by dmesg.
tsz will not receive the data from the client, unless tsz opens tty to read. And we have to make sure that dmesg ( or other programs ) do not read any data from stdin, otherwise the data received by tsz may be incomplete.
In other words, it would be difficult to support stdin, and it can only be used in limited scenarios.


Use logrus for logging http server panics

DO NOT MERGE - i want to include the norman pr that updates controller logging as a separate commit in this pr
http.Server will recover from panics and log the panic's goroutine
dump to its ErrorLogger. By default, the error logger uses the
log package's builtin logger, which typically just logs to stderr.
However, when running rancher in "embedded" mode, kubernetes code
runs which reconfigures the builtin logger to use glog. In this case,
glog is just writing to a file in the /tmp dir, which was causing us
to not print panics.

LGTM


Fix multiarch build permissions

Just trying to keep things moving.
This is a follow up to https://github.com/jenkinsci/docker/pull/750 to enable multiarch builds.  Related to https://github.com/jenkinsci/docker/issues/646 and https://github.com/jenkinsci/docker/issues/686, there is currently a permission issue that is inhibiting pushing the multiarch images to the jenkins4eval repo.
It sounds like the Dockerhub user that is being used to build and push the images needs to have permissions to the jenkins4eval repo.
Job results: https://ci.jenkins.io/job/Packaging/job/docker/job/jenkins4eval/
/cc @oleg-nenashev @imdurgadas @batmat @MarkEWaite

Any movement on this @jmreicha @oleg-nenashev?

@lag-linaro test PR open here https://github.com/jenkinsci/docker/pull/807
Basically, I think there are some authentication issues that need to be debugged.

I can help look into this, we need to find out what the withDockerCredentials is using and make sure the jenkins4eval has the correct permissions. According to @oleg-nenashev it should since jenkinsfile-runner is able to publish to jenkins4eval just fine.

Should be able to run the build/publish script with the debug flag turned on to see what's happening.

The docker push already has --log-level debug, regardless of the debug flag passed to the publish script:
https://github.com/jenkinsci/docker/blob/jenkins4eval/publish-experimental.sh#L222
https://github.com/jenkinsci/docker/blob/jenkins4eval/publish-experimental.sh#L260
I am not sure calling the script with -d will help much in getting more information. Are you proposing adding -D to the docker call as well to enable debug mode?

Add -d to the publish script, e.g. https://github.com/jenkinsci/docker/blob/jenkins4eval/Makefile#L57.

I am not sure that will provide information as to why the docker command is failing (for permissions), but we can definitely try it.

@jmreicha  See https://github.com/jenkinsci/docker/pull/855

It doesn't seem like my changes did anything during the actual publish phase

Ok, this build went though, but docker info isn't showing the username. Also, it looks like DOCKER_CONFIG exists since there wasn't an early exit from the script with an error message. Other ideas?

I'm guessing the username being empty is the problem, but I'm not really sure why the credentials are being set.

I believe this is fixed now. The Packaging folder on ci.jenkins.io had it's own credentials setup that were wrong, these were inherited by the docker builds. That issue has been fixed and pushing to docker hub should work now.


Missing values in LUC files

Originally opened by @dariak-bsc in #31

I ran the checker with the mask taken not from the reference file but from the corresponding variable in the file which is being checked, e.g. if I am checking the variable c3ann in the file multiple-states_input4MIPs_landState_RESCUE_PIK-MAgPIE-4-7-C-SSP2EU-NPi-2024.4.25_gn_2015-2100.nc then I am taking the missing values in c3ann at the first timestep as the mask to check the following timesteps. And in this case, I am not getting any missing values at the following timesteps. When I take the mask from the reference file, then I get missing values.

Is it fair to say (@dariak-bsc / @etiennesky / @pascal-sauer) that the issue here is what grid cells are considered land vs. ocean?
Here is the map showing the delta between the reference data and the forcings land use mask:

@dariak-bsc just for clarity, can you tell us what files precisely are being compared in the above image?

yes it seems like the issue is at the border of oceans and inner lakes, so perhaps it's the remapping method that needs minor tweaking (e.g. using nearest neighbor or a binary land-sea mask)

I'm working on a nearest neighbor approach

@dariak-bsc just for clarity, can you tell us what files precisely are being compared in the above image?

Hi @gidden , for this image, I took the files on the ftp:

(1) esarchive_ftp/rescue/forcings/land_use/2024-04-25/multiple-states_input4MIPs_landState_RESCUE_PIK-MAgPIE-4-7-C-RESCUE-dir-v4p0-EocBudg1150-OAE-off-2024.4.25_gn_2015-2100.nc
(2) esarchive_ftp/rescue/reference/landuse/multiple-states_input4MIPs_landState_CMIP_UofMD-landState-2-1-h_gn_2015.nc

I extracted the masks as the mask for c3ann at the first timestep for both files, and then calculated the difference between the mask for the file (1) and the mask for the file (2).
I did the same for the historical file provided by @etiennesky but it was equal to the mask for the file (2).

@pascal-sauer I guess the newest version fixed the reported issue?

yes, it is fixed now


Update CI/CL, drop Python 3.7

Python 3.7 is EOL, let's at least use 3.8 as the minimum.

Merging since the failing tests are complaining about Cython code, something out of scope for this commit.

I'd rather we didn't merge PRs when CI is failing. If we have flakey tests.. fix them!

I'd rather we "fail-forward" instead of making one mega-commit that fixes everything