A way to configure scale up policy / limit the number of nodes added in a window of time

Description
What problem are you trying to solve?
Karpenter does not have a way to configure a scaleup policy or limit the number of nodes added in a window of time. This allows Karpenter to ask for more nodes than the control plane can actually handle. If this could be configured or limited in some way in a central way by Karpenter this could allow protection for the control plane against run-away scaling.
There's also further information about these limitations documented in https://aws.github.io/aws-eks-best-practices/scalability/docs/control-plane/#limit-workload-and-node-bursting :

To avoid reaching API limits on the control plane you should limit scaling spikes that increase cluster size by double digit percentages at a time (e.g. 1000 nodes to 1100 nodes or 4000 to 4500 pods at once).

So this feature is in a way asking karpenter to support enforcing the very behavior EKS documentation suggests is best practice.
How important is this feature to you?
Twilio Segment has recently had 3 high severity incidents recently which all were caused by large scaling up events leading to cascading failures on the EKS control plane, which actually resulted in the Horizontal Pod Autoscaler asking for even more pods. Of course this exaserbated the situation. It's very important that these Large Scale Events (LSEs) / incidents do not occur again for obvious financial reasons.
As a result one of the many action items is to add HPA scaleup policies such as those mentioned in the documentation. Unfortunately a pod scale-up policy is only a proxy for what is actually important to limit in this case, which is the number of nodes to scale up in a window of time.

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Kubernetes scalability tuning is a difficult, multi-component, system integration problem. In this case, Karpenter sits in the middle, between the work (pods) and the capacity (nodes). I'm a bit cautious about growing Karpenter's API surface to include performance considerations of downstream services, as those performance profile and integration points will change over time.

To avoid reaching API limits on the control plane you should limit scaling spikes that increase cluster size by double digit percentages at a time (e.g. 1000 nodes to 1100 nodes or 4000 to 4500 pods at once).

This recommendation is targeting broad Kubernetes use cases, where clusters may not always be reactively autoscaled. The guidance around 1000 -> 1100 nodes is targeted at use cases where customers are configuring node scaling directly, like with EC2 Autoscaling Groups or Managed Node Groups. The latter pod numbers 4000 -> 4500 are more relevant for Karpenter.
Then the question becomes:
"How can I rate limit the number of pods joining my cluster, to avoid impacting the control plane?".
This is actually a very good question, and ideally something that's provided automatically with something like https://kubernetes.io/docs/concepts/cluster-administration/flow-control/. Let me connect with the EKS Scalability team, and see what the guidance is.

I don't know if Twilio Segment is speaking directly with EKS scalability team specifically but we are in support channels with AWS. They provided those docs to us as well as the flow-control one. They've also taken steps to increase the base size of the control plane nodes. There are more actions being taken, but I think that is digressing a bit too far from the original request.
Flow control and priority still require k8s nodes to successfully join the cluster. If nodes can't join the cluster because a majority of new nodes can't start up cilium or whatever alternative cni plugin is used to allocate IP addresses, because they are receiving 429 rate limit errors from the API server, and the API servers are crashing... I'm not sure it would always be enough to alleviate all the problems, maybe more limit the blast radius (which is valuable to some extent). It still seems like being able to configure some rate / policy of how quickly new nodes join the cluster could more drastically help the situation or completely protect the Control Plane.
Specifically "How can I rate limit the number of pods joining my cluster, to avoid impacting the control plane?" probably would be answered by configuring a pod scale up policy as documented in https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior . Like I said, we are taking steps to add these for all HPA configurations... but any calculation made about what the scale-up policy should be is derived from the percentage of nodes for these pods or number of nodes currently in the cluster. It just feels like we're dancing around the actual limitation, the nodes.

Let me connect with the EKS Scalability team, and see what the guidance is.

Let me know what they say, happy to talk through our TAM to schedule a call with all parties if that would help.

Hello Elis,
Thank you for the detailed explanation. Everything you're saying makes sense. Let me go back to the Twilio Segment compute team and discuss this with them further. I'm part of the Segment Release Engineering team and was not in the room with AWS when discussions on this topic with AWS support / internal AWS teams occurred. I was updated today that daemon set limits were more impactful than the scale up policy was originally suggested before this data was either discovered by them or in collaboration with AWS. That lines up a lot with everything you wrote. Let me get back to you shortly. Thank you again.

Hello Ellis,
The segment compute team has requested through our TAM to meet with the karpenter team and discuss scaling recommendations further. For now feel free to close this github issue.
Thank you,
-Asaf


UI changes for Peoples page

Would it be better to create little profiles on the site - image and a paragraph bio - these could then have a link out to the longer bios.

Sush, that is what I was thinking, I think it is the way to go. That way longer bios can be added too over time.


[Avail] Add support for Avail testnet (Kate)

** Please name the issue/PR after this format**
Add support for Kate testnet on Avail - a Substrate-based blockchain

[x] Viewing balance
Transfering & Staking: improve later (https://github.com/Koniverse/SubWallet-ChainList/issues/75)

If this is your first time integrating, the following information is required

A short introduction about your project/ your team: Avail is a Substrate-based blockchain founded by Anurag - one of Polygon co-founders (https://www.availproject.org/)
Social media links (telegram, discord, website, linkedin,...)
Twitter: https://twitter.com/AvailProject
Discord: https://discord.com/invite/y6fHnxZQX8
Github: https://github.com/availproject

If you want to integrate a chain, you would need to provide

Name of the blockchain: Avail Kate Testnet
Type of the blockchain (can be Substrate, EVM or both): Substrate
Explorer: https://kate.avail.tools/
RPC Endpoint: https://kate.avail.tools/v1
WS Endpoint: wss://kate.avail.tools/ws
Chain Specification: chainspec.raw.json
Chain Info: chaininfo.txt
Docs: https://availproject.github.io/join-the-network/networks
API changes: https://github.com/availproject/avail/blob/da91630d5c472ff738058698dd8e5b7cf4694ea2/tests/js_api/src/api.ts#L8-L134
Logo:

Have some problem with sign transaction.
It will be resolved in https://github.com/Koniverse/SubWallet-Extension/issues/1222

Resolved in https://github.com/Koniverse/SubWallet-Extension/issues/1541


steamVR doesn't find any of my components

I had a set up of my CV1 + sensor with two psmove controllers and two cameras, it worked great. Alot better then i had expected. But i tried to play a few games and all of them show that nothing is connected. I tried reinstalling drivers and steamvr itself and to avail. Help please

Fixed it by uninstalling steam client and reisntalling


Fixing issue-98

Fixing ability to attach files from disk

@anto-wahanda thank for quick fix 👍
LGTM.

No problem. @BorisOsipov I hope the test is ok.

Just to confirm I have also tested this in the sample project I put together to demonstrate the issue and I've also published a npm package to our internal repo to demonstrate it works with the actual project I want to use this for.


UI - platform web initial form work

Issue:

bcgov/entity/issues/

Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the namex license (Apache 2.0).

I'm happy to merge this now just commenting to keep track of things.
Looks pretty good just some small things like i18n/aria-labels, styling

form/common/crainfo aria labels need i18n
form/common/ReviewRow should the divs still be w-1/3 when flex direction = flex-col
form/host/PrincipalResidence most aria-labels need i18n
form/host/property need aria labels on ConnectSection components and aria-label i18n
form/host/ReviewConfirm img alt text needs i18n, 'Undeclared' needs i18n
form/platform/Details help text wont be rendered on ConnectField components when wrap with form grp prop = false
useWindowSize from vueuse instead of custom useScreenSize composable
missing i18n in stores/hostProperty
fee widget, increase z-index (button icon in buttonControl button is showing through the fee widget heading when fee widget is collapsed on small screen), make heading focusable (and styling) when floating at the bottom on small screen, remove focus option when on large screen, currently fee widget covers the footer on small screens, maybe we could move the fee widget to be above the button control component on small screens? will need to think about this one)
stepping through the from when a form is invalid causes a uncaught form validation error in the console
something is causing the screen width to overflow under ~1370px (fee widget width?)
t('createAccount.propertyForm.platformUrl') 'platform' typo

I'm happy to merge this now just commenting to keep track of things.
Looks pretty good just some small things like i18n/aria-labels, styling

form/common/crainfo aria labels need i18n
form/common/ReviewRow should the divs still be w-1/3 when flex direction = flex-col
form/host/PrincipalResidence most aria-labels need i18n
form/host/property need aria labels on ConnectSection components and aria-label i18n
form/host/ReviewConfirm img alt text needs i18n, 'Undeclared' needs i18n
form/platform/Details help text wont be rendered on ConnectField components when wrap with form grp prop = false
useWindowSize from vueuse instead of custom useScreenSize composable
missing i18n in stores/hostProperty
fee widget, increase z-index (button icon in buttonControl button is showing through the fee widget heading when fee widget is collapsed on small screen), make heading focusable (and styling) when floating at the bottom on small screen, remove focus option when on large screen, currently fee widget covers the footer on small screens, maybe we could move the fee widget to be above the button control component on small screens? will need to think about this one)
stepping through the from when a form is invalid causes an uncaught form validation error in the console
something is causing the screen width to overflow under ~1370px (fee widget width?)
t('createAccount.propertyForm.platformUrl') 'platform' typo
form padding/margin is uneven on small screens

Yeah sounds good nice comments, I'll try to get most of these and then merge later today so that its easier for you to get started. I'll be leaving all the 'property manager / host' stuff alone since we're not doing it now (maybe we'll go back to it later)


Cold tier details

The section quoted below does not mention the newly GA cold tier

Consider the following information when using access tiers:

Only the Hot and Cool access tiers can be set at the account level. The Archive access tier isn't available at the account level.
Hot, Cool, and Archive tiers can all be set at the blob level during upload or after upload.
Data in the Cool tier has slightly lower availability, but offers the same high durability, retrieval latency, and throughput characteristics as Hot tier data. For data in the Cool tier, slightly lower availability and higher access costs can be acceptable trade-offs for lower overall storage costs compared to the Hot tier.
Archive storage stores data offline and offers the lowest storage costs. However, it also carries the highest data rehydration and access costs.
For more information, see Hot, Cool and Archive access tiers for blob data.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 392e90a0-af77-999e-39a1-78f09ffe0d06
Version Independent ID: 411ca9a3-4370-f9ca-f14c-5385e13d9075
Content: Access tiers for blob data - Azure Storage
Content Source: articles/storage/blobs/access-tiers-overview.md
Service: storage
GitHub Login: @normesta
Microsoft Alias: normesta

@thommck
Thanks for your feedback! We will investigate and update as appropriate.

Thank you for raising this @thommck. This block of text doesn't appear to exist in this article (https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview).  Would you mind directing me to the article where you found the list that you present above? Thank you!


add .gitignore, add initial Kotlin framework

This pull might add .gitignore, file has ignores for linux, windows, and macos, need to add explicit xcode support if we need it (we should not need xcode)

pull request should be updated to add files for initial commit for kotlin support. Enough files should be committed so that you might run the android emulator in android studio on the default device, using the sample app that was pasted from https://www.jetbrains.com/help/kotlin-multiplatform-dev/compose-multiplatform-new-project.html#support-user-input.
Let me know of any errors you run into while trying to build, Gradle refused to run the app until I rebuilt the whole project after a while.


bump: [#4684] Update sinon dependency to latest version

Addresses #4684
#minor
Description
This PR updates the sinon package to the latest version 19.0.2 and @types/sinon to version 17.0.3.
Specific Changes

Updated sinon from 9.2.4 and 2.1.0 to the latest 19.0.2 in the root's package.json and tools folder.
Updated @types/sinon from 9.0.11 to 17.0.3.

Testing
The following images show the installed version and the unit tests passing.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.4%) to 84.997%

Totals

Change from base Build<PHONE_NUMBER>6:
-0.4%

Covered Lines:
20351

Relevant Lines:
22910

💛  - Coveralls


Understanding addImages method

I'm new to iOS and likely going to use this library, trying to understand addImages method implementation, could see that add kitten object is added using main thread? why not directly add in it async block like mentioned below?
addImages {
....
kitten.imageURL = kittenURLs[kittenIdx];
kitten.imageSize = CGSizeMake(width / scale, height / scale);
//                dispatch_sync(dispatch_get_main_queue(), ^{
//                    [tmpKittens addObject:kitten];
//                });
[tmpKittens addObject:kitten];
}

It's just out of laziness. If we do everything on one thread, the main thread in this case, we don't have to worry about thread safety. Otherwise we would have had to create some kind of thread synchronization method to make sure the asynchronous operations didn't happen at the same time and clobber tmpKittens.

Thanks Garrett. Got it.


Correct TypeScript Typing for setNotificationListener Callback

Summary:
This pull request addresses a discrepancy in the TypeScript type definition for Pushy's setNotificationListener method. The current type definition does not explicitly indicate that the callback function can return a Promise, which contradicts the example provided in the Pushy documentation under the "Listen for notifications" section.
Pushy.setNotificationListener(async data => {
   // implementation
});

Changes:

Updated the Pushy.d.ts file to reflect that the setNotificationListener method's callback argument is an asynchronous function, thereby returning a Promise.

Justification:
Aligns TypeScript definitions with Pushy documentation, which suggests using an async function as the callback, enhancing developer experience and type accuracy.

Hi @berdimyradov,
Thanks so much for bringing this to our attention.
Merged & published to npm in 1.0.36.
Thanks for your contribution! 💯


Consul 1.0.1 breaks reading the configmap config

When running consul 1.0.1 I get the following error attempting to spin up consul using the artifacts in this tutorial
Percival-Derpington:manifests mattcase$ kubectl logs consul-0
==> config: ReadFile failed on /consul/config/..data: read /consul/config/..data: is a directory

Using the exact same configmap but utilizing 1.0.0 results in proper startup:
Percival-Derpington:manifests mattcase$ kubectl logs consul-0
==> Starting Consul agent...
bootstrap_expect > 0: expecting 3 servers
==> Consul agent running!
           Version: 'v1.0.0'
           Node ID: '4a369f12-a478-529f-c426-07db9baa7ac0'
         Node name: 'consul-0'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [<IP_ADDRESS>] (HTTP: 8500, HTTPS: 8443, DNS: 8600)
      Cluster Addr: <IP_ADDRESS> (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true

it's odd because it's configured via -config-file=/consul/config/server.json in the manifest for the StatefulSet, and ..data is in fact a directory pointing to the mounted configmap. When investigating in the running 1.0.0 container I see the following:
/consul/config # ls -lha
total 12
drwxrwsrwx    3 consul   consul      4.0K Nov 21 17:03 .
drwxr-xr-x    4 consul   consul      4.0K Oct 31 18:24 ..
drwxr-sr-x    2 root     consul      4.0K Nov 21 17:03 ..119811_21_11_17_03_00.120617316
lrwxrwxrwx    1 root     root          33 Nov 21 17:03 ..data -> ..119811_21_11_17_03_00.120617316
lrwxrwxrwx    1 root     root          18 Nov 21 17:03 server.json -> ..data/server.json

So I'm not sure why it's trying to read ..data as a file when it's being directed specifically to read server.json

I have the same problem for v1.0.2

@fastest963
Im using this config to deploy consul on kubernetes.
cat deployments/consul-1.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: consul-1
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
app: consul
member: consul-1
spec:
securityContext:
fsGroup: 1000
containers:
- name: consul
image: "consul:1.0.1"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- "consul"
- "agent"
- "-advertise=<IP_ADDRESS>"
- "-rejoin"
- "-bind=<IP_ADDRESS>"
- "-ui"
- "-node=consul-1"
- "-server"
- "-bootstrap-expect=3"
- "-raft-protocol=3"
- "-retry-join=kubernetes.default.svc.cluster.local"
- "-retry-join=kubernetes.default.svc.cluster.local"
- "-retry-join=kubernetes.default.svc.cluster.local"
- "-client=<IP_ADDRESS>"
- "-data-dir=/consul/data/"
- "-domain=cluster.local"
- "-disable-host-node-id"
ports:
- containerPort: 8300
name: server
- containerPort: 8301
name: serflan-tcp
protocol: "TCP"
- containerPort: 8301
name: serflan-udp
protocol: "UDP"
- containerPort: 8302
name: serfwan-tcp
protocol: "TCP"
- containerPort: 8302
name: serfwan-udp
protocol: "UDP"
- containerPort: 8400
name: rpc-port
- containerPort: 8500
name: api-port
- containerPort: 8600
name: consuldns-tcp
protocol: "TCP"
- containerPort: 8600
name: consuldns-udp
protocol: "UDP"
volumeMounts:
- name: "consulconfig"
mountPath: /etc/consul
volumes:
- name: "consulconfig"
"secret": {
"secretName": "consul-config"
}
but im facing issue
HTTP error code from Consul: 500 Internal Server Error
Error message from Consul: No cluster leader
Consul returned an error. You may have visited a URL that is loading an unknown resource, so you can try going back to the root. If your ACL token was not found you can reset it, and then you will be redirected to the settings page to enter a new ACL token.
2018/05/24 08:35:29 [ERR] agent: failed to sync remote state: No cluster leader
2018/05/24 08:35:33 [ERR] agent: Coordinate update error: No cluster leader
2018/05/24 08:35:37 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:

Failed to join <IP_ADDRESS>: dial tcp <IP_ADDRESS>:8301: i/o timeout

Failed to join <IP_ADDRESS>: dial tcp <IP_ADDRESS>:8301: i/o timeout

Failed to join <IP_ADDRESS>: dial tcp <IP_ADDRESS>:8301: i/o timeout
2018/05/24 08:35:37 [WARN] agent: Join LAN failed: , retrying in 30s
2018/05/24 08:36:01 [ERR] agent: failed to sync remote state: No cluster leader
2018/05/24 08:36:05 [ERR] agent: Coordinate update error: No cluster leader
2018/05/24 08:36:07 [INFO] agent: (LAN) joining: [kubernetes.default.svc.cluster.local kubernetes.default.svc.cluster.local kubernetes.default.svc.cluster.local]
2018/05/24 08:36:18 [ERR] http: Request GET /v1/coordinate/nodes?dc=dc1&token=, error: No cluster leader from=<IP_ADDRESS>:37920
2018/05/24 08:36:23 [ERR] agent: failed to sync remote state: No cluster leader
2018/05/24 08:36:37 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:

Failed to join <IP_ADDRESS>: dial tcp <IP_ADDRESS>:8301: i/o timeout

Failed to join <IP_ADDRESS>: dial tcp <IP_ADDRESS>:8301: i/o timeout

Against 1.2.0 you get a similar issue regarding the docker entrypoint attempting to chmod /consul/config and failing due to it being a read-only FS.  #34 resolves this.

I'm now using a different directory. This should be now fixed on master.


Moto tests failing; can't find eu-south-1 region

This is sort of a follow up to https://github.com/spulec/moto/issues/2789. I was running into the problem reported in that issue, and followed the suggestion to upgrade to 1.3.15.dev550+
I've upgraded to moto-1.3.15.dev751, and now I'm running into an error related to what seems to be a region map or configuration that assumes the presence of the eu-south-1 region. The stack trace is reproduced below.
Traceback (most recent call last):
  File "/bin/test-package.py", line 5, in <module>
    import moto
  File "/build/lib/python2.7/site-packages/moto/__init__.py", line 3, in <module>
    from .acm import mock_acm  # noqa
  File "/build/lib/python2.7/site-packages/moto/acm/__init__.py", line 2, in <module>
    from .models import acm_backends
  File "/build/lib/python2.7/site-packages/moto/acm/models.py", line 7, in <module>
    from moto.ec2 import ec2_backends
  File "/build/lib/python2.7/site-packages/moto/ec2/__init__.py", line 2, in <module>
    from .models import ec2_backends
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 5188, in <module>
    region.name: EC2Backend(region.name) for region in RegionsAndZonesBackend.regions
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 5188, in <dictcomp>
    region.name: EC2Backend(region.name) for region in RegionsAndZonesBackend.regions
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 5101, in __init__
    super(EC2Backend, self).__init__()
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 858, in __init__
    super(InstanceBackend, self).__init__()
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 1140, in __init__
    super(TagBackend, self).__init__()
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 2386, in __init__
    super(EBSBackend, self).__init__()
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 1922, in __init__
    super(SecurityGroupBackend, self).__init__()
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 1368, in __init__
    self._load_amis()
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 1375, in _load_amis
    self.amis[ami_id] = Ami(self, **ami)
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 1326, in __init__
    volume = self.ec2_backend.create_volume(15, region_name)
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 2390, in create_volume
    zone = self.get_zone_by_name(zone_name)
  File "/build/lib/python2.7/site-packages/moto/ec2/models.py", line 1731, in get_zone_by_name
    for zone in self.zones[self.region_name]:
KeyError: u'eu-south-1'
:::: src: FAILED
BUILD FAILED

I then upgrade my version of botocore to 1.17.45, but I'm still running into the same issue.
Please advise.

Hi @jjbuck, the eu-south--region was only made available as of moto >= 1.3.15.dev772.
Let me know if upgrading solves both your problems.
FYI: The region-issue happens whenever botocore adds a new region that is not yet recognized by moto. The workaround (until moto supports the region) is typically to downgrade to a version of botocore that doesn't have the new region yet.

Going to close this due to a lack of response, but feel free to let me know if this is still an issue.


Convert the rest of the STs to python.

A general comment: most of these tests just convert the original bash scripts to python where you essentially convert the commands to host.execute lines.
I'd much rather see some reusable, high level python commands that eventually map down to the command line calls rather than putting the CLI syntax directly into all our STs.  I think we'll find this much harder to maintain and build on in the long run.

The goal of the first conversion was basically just that (decided by a conversation with Tom and someone else I can't remember). I come from a mindset of wanting to build as much structure as possible, but that's from my sense of aesthetics and not from practicality. So in this case I tried to push myself pretty hard to not make infrastructure, in order to practice being agile and usefulness-based. So I have a pile of things I want to do to improve these tests, but I'm unclear on where to aim along that tradeoff.
That said, it doesn't seem like people are really using the tests all that much. Most of the ST runs are mine. People haven't been wanting to write tests while I've been working on them. So I think we should merge this at it's current level of complexity, and if people want more structure I will gladly implement it.

@alexaltair I'm fine merging without infrastructure if that's a trade off we're making from an agile standpoint.  But, I'd like to see that infrastructure in place before we go a lot further on writing additional tests---writing new tests in this current style is going to make more work when we add new structure.
ST runs should become part of our continuous integration infrastructure so they get run all the time by everyone.


Using google.cloud.operation for GAPIC Speech async?

I've been struggling today with the right way to implement this.
I can't seem to make_secure_stub for speech.
This is the error I get...
File "/Users/daspecster/Documents/google/google-cloud-py-2/.tox/py27/lib/python2.7/site-packages/google_cloud_speech-0.20.0.dev0-py2.7.egg/google/cloud/speech/_gax.py", line 92, in async_recognize
    OPERATIONS_API_HOST)
  File "/Users/daspecster/Documents/google/google-cloud-py-2/.tox/py27/lib/python2.7/site-packages/google_cloud_core-0.20.0-py2.7.egg/google/cloud/_helpers.py", line 642, in make_secure_stub
    transport_creds = grpc.ssl_channel_credentials()
AttributeError: 'module' object has no attribute 'ssl_channel_credentials'

Example code:
        config = RecognitionConfig(
            encoding=sample.encoding, sample_rate=sample.sample_rate,
            language_code=language_code, max_alternatives=max_alternatives,
            profanity_filter=profanity_filter,
            speech_context=SpeechContext(phrases=speech_context))

        audio = RecognitionAudio(content=sample.content,
                                 uri=sample.source_uri)
        self._client._operations_stub = make_secure_stub(
            self._client.connection.credentials,
            DEFAULT_USER_AGENT,
            operations_grpc.OperationsStub,
            OPERATIONS_API_HOST)

        api = self._gapic_api
        operation = api.sync_recognize(config=config, audio=audio)
        return Operation.from_pb(operation, self._client)

Looking at the sample example it seems like the channel creation is different than it is in our _helpers.make_secure_stub

This seems to work though.
╰─$ .tox/py27/bin/python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
Type "help", "copyright", "credits" or "license" for more information.
>>> import grpc
>>> grpc.ssl_channel_credentials()
<grpc.ChannelCredentials object at 0x109463950>

@bjwatson could something be messing up the namespace in
    'gapic-google-cloud-speech-v1beta1 >= 0.11.1, < 0.12.0',
    'grpc-google-cloud-speech-v1beta1 >= 0.11.1, < 0.12.0',

Also, @bjwatson, is gapic-google-longrunning ok to use or is it not really production ready yet?

@daspecster It seems like there is an old version of grpc? Also, you don't need make_secure_stub for Speech (not used in Pub/Sub or Logging).
There is a "bug" in that we should pass credentials through, but you shouldn't worry about handling it to get Speech in.

@dhermes, Pub/Sub and Logging, don't use Operations AFAICT.
A channel has to be made in order to make the OperationStub.
https://github.com/GoogleCloudPlatform/python-docs-samples/blob/2da1c09b056798dd54be7106010f9d5610952bef/speech/grpc/transcribe_async.py#L32-L51
I could just do that but that's not very DRY since we have it in core._helpers right?

Fixed with #2632.

@daspecster Yes, gapic-google-longrunning is okay to use, although I plan to merge it into the next release of google-gax (with the same path and module name).
We are also designing a more convenient layer for you to use in the future that will sit on top of the gapic-google-longrunning code. I'll take a look at what you did for Speech, and want to discuss with you what pain points you had.

Ok sure! Thanks @bjwatson!


[TC-PRS-1.1] Global Attributes Missing From Pressure Measurement Cluster

Problem
The following global cluster attributes seem to be missing in the Pressure Measurement cluster

FeatureMap
AttributeList
CommandList
EventList

Spec Ref

YAML Test Script Logs

The following global attributes seem to be missing in the all clusters

FeatureMap
AttributeList
EventList
ClientGeneratedCommandList
ServerGeneratedCommandList

The following Testcases are missing the global attribute
TC-BI-1.1 ,TC-BOOL-1.1 ,TC-CC-1.1 ,TC-DIAGTH-1.1 ,TC-EMR-1.1 ,TC-ILL-1.1 ,TC-LVL-1.1 ,TC-MC-1.1, TC-OCC-1.1 ,TC-OO-1.1 ,TC-PCC-1.1, TC-PRS-1.1 ,TC-RH-1.1, TC-TM-1.1, TC-TSTAT-1.1, TC-TSUIC-1.1, TC-WNCV-1.1
Spec screenshot:

@kvikrambhat is this still an issue?

@holbrookt we are still facing attribute missing issue
Spec screenshot:

Execution screenshot:

On current tip I see:
./out/debug/standalone/chip-tool pressuremeasurement read
[1644466642455] [32667:14897110] CHIP: [TOO] Missing attribute name
Usage:
  ./out/debug/standalone/chip-tool pressuremeasurement read attribute-name [param1 param2 ...]

  +-------------------------------------------------------------------------------------+
  | Attributes:                                                                         |
  +-------------------------------------------------------------------------------------+
  | * measured-value                                                                    |
  | * min-measured-value                                                                |
  | * max-measured-value                                                                |
  | * attribute-list                                                                    |
  | * cluster-revision                                                                  |
  +-------------------------------------------------------------------------------------+

No FeatureMap, which seems like a mismatch against current spec.
No EventList; that's tracked in https://github.com/project-chip/connectedhomeip/issues/15011.

Currently chip-tool generates commands for all the attributes/commands for all clusters.
In theory, if it is missing it is an SDK bug, not a test harness issue.
Here is the output of chip-tool on ToT. You can see that it still miss event-list but this is because of #15011 as mentioned by Boris.
So I'm closing this issue, assuming event-list will automatically appears in chip-tool once #15011 is resolved.
$ ./out/debug/standalone/chip-tool pressuremeasurement read

Usage:
  ./out/debug/standalone/chip-tool pressuremeasurement read attribute-name [param1 param2 ...]

  +-------------------------------------------------------------------------------------+
  | Attributes:                                                                         |
  +-------------------------------------------------------------------------------------+
  | * measured-value                                                                    |
  | * min-measured-value                                                                |
  | * max-measured-value                                                                |
  | * tolerance                                                                         |
  | * scaled-value                                                                      |
  | * min-scaled-value                                                                  |
  | * max-scaled-value                                                                  |
  | * scaled-tolerance                                                                  |
  | * scale                                                                             |
  | * server-generated-command-list                                                     |
  | * client-generated-command-list                                                     |
  | * attribute-list                                                                    |
  | * feature-map                                                                       |
  | * cluster-revision                                                                  |
  +-------------------------------------------------------------------------------------+


misleading "Timeout getting IP address" when host memory is low

Rancher versions:
rancher/server: v1.6.10 (separate Host)
rancher/agent: v1.2.6 (2 Hosts)
Infrastructure Stack versions:
healthcheck: v0.3.3 (catalog v0.3.3)
ipsec: net:v0.11.9 (catalog v0.1.4)
network-services: mgr v0.7.8 / metadata v0.9.4  (catalog v0.2.6)
scheduler: v0.8.2 (catalog v0.6.3)
kubernetes (if applicable):
Docker version: (docker version,docker info preferred)
`Containers: 89
Running: 62
Paused: 0
Stopped: 27
Images: 370
Server Version: 17.06.2-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 971
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local rancher-nfs
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.71GiB
Name: host1
ID: JH47:ITJ3:WKUV:AEBS:RMQI:KMH6:OSSI:Z4KU:RKAW:2SYZ:OL5X:ROP5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
<IP_ADDRESS>/8
Live Restore Enabled: false
WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
`
`Containers: 72
Running: 36
Paused: 0
Stopped: 36
Images: 82
Server Version: 17.06.2-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 443
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local rancher-nfs
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.71GiB
Name: host2
ID: JH47:ITJ3:WKUV:AEBS:RMQI:KMH6:OSSI:Z4KU:RKAW:2SYZ:OL5X:ROP5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Live Restore Enabled: false
WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
`
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
host1 and host2:
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
3.16.0-4-amd64
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
VSphere
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
1 separate rancher server, 2 hosts in environment, DB(s) internal
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:
Our system has been running for about half a year now, upgrades from earlier rancher versions and docker included. We recently started to add automated deployment from gitlab-ci via rancher-cli (rancher up + compose files) and running builds via gitlab-multi-runner the same rancher environment. After some time we hit the "aufs cache" problem of docker 17.06.1 where old diffs wouldn't be cleaned from /var/lib/docker/aufs, an upgrade to docker 17.06.2 solved that but it seems there are other problems turning up with extended use. We are currently running about 100 containers, the memory of both hosts is at around 9/16GiB and 14,5/16GiB.
I just tried adding another container that takes up to 3 GB of memory on host2. I saw several containers turn "unhealthy" and finally be restarted. That's when the shit hit the fan ... response times from rancher scheduler increased and ran into timeouts, finally none of the unhealthy containers came back online, saying "Timeout getting IP address" ... kind of distracting from the memory issue of the host. Some status bar in the hosts tab or general warning/error notification would be great.
After removing the new container, everything turned back to normal again.
Results:

After adding a memory limit, we also saw containers using swap up to 10 GB of 4 GB available instead. Again some global monitoring Dashboard would be great.

Thanks for reporting, closing this in favor of https://github.com/rancher/rancher/issues/3527


[InputBase][material-ui] Add sx type to  input , root slot

Styles applied through sx prop are appplied to elements but sx type was not added. This PR adds sx type
Before : https://codesandbox.io/s/nice-james-yqp8vr?file=/src/Demo.tsx
After: https://codesandbox.io/s/jovial-snyder-ww9dd5?file=/src/Demo.tsx

[x] I have followed (at least) the PR section of the contributing guide.

I believe this change could be applied to all components with slots and slotProps.

What do you think better action plan for this, fix all ts errors in 1 PR or individual PR for each component

I believe this change could be applied to all components with slots and slotProps.

What do you think better action plan for this, fix all ts errors in 1 PR or individual PR for each component ?

Let's not do this now. There will be lot of PRs. This is fine for now. We can tackle others when someone needs it.


Terraform parser not working on OSX and Linux

When running locally either using the binary or the docker image I get below error where it can't find the library "libace-terraform-parser-linux.dll"
docker run -e AZURE_CLIENT_ID=$ARM_CLIENT_ID -e AZURE_TENANT_ID=$ARM_TENANT_ID -e AZURE_CLIENT_SECRET=$ARM_CLIENT_SECRET -v .:/app/templates thecloudtheory/azure-cost-estimator templates/terraform.tf $ARM_SUBSCRIPTION_ID [ Redacted ]
                                          .***      ***
                               ,    ***.********,****
                                 **********************
                           *    ************. ***********  *
                         ///   *******, ****  ////////////******
                           .//************ ///// /////////********
                       ///////// ********///////// //////.********    (,
                 .   ///////////// **** ////////////,////******* ( (/
                   *////////////////   ////////////////  ** ((((((((((((
                    ///,  */////////.   **************,    ((((((((((((((
                   ///////////////.(((((((,******** ///// (((((( (((((((((
                   ///////////// ((((((((((((   ////////// ((((((((((*((((
                    ////////// ((((((((((( ((    //////////(((((((((((((((
                //   /////// (((((( (((((((((((((((( ///////(((((((((((((
                   //   /  ( ((((((((((((((((,((((((((((.////(((((((((  ((

                              ACE (Azure Cost Estimator) [1.2.0]

General help: https://github.com/TheCloudTheory/arm-estimator/discussions
Bugs & issues: https://github.com/TheCloudTheory/arm-estimator/issues

Run configuration:
-> Scope: ResourceGroup
-> Susbcription Id: [Redacted]
-> Resource group: [Redactoed]
-> Template file: templates/terraform.tf
-> Deployment mode: Incremental
-> Threshold: Not Set
-> Parameters file: Not Set
-> Currency: USD
-> Generate JSON output: False
-> Silent mode: False
-> Redirect stdout: False
-> Disabled detailed metrics: False
-> Generate HTML output: False
-> HTML output filename: Not Set
-> Dry run enabled: False
-> Cache disabled: False

Starting parsing using embedded Go parser.
Unhandled exception: System.DllNotFoundException: Unable to load shared library 'ace-terraform-parser-linux.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libace-terraform-parser-linux.dll: cannot open shared object file: No such file or directory
at ACE.Compilation.TerraformCompiler.GenerateParsedPlan(GoString workingDir, GoString planFile)
at ACE.Compilation.TerraformCompiler.Compile(FileInfo templateFile) in /src/ace/Compilation/TerraformCompiler.cs:line 52
at ACE.Compilation.TemplateCompiler.Compile() in /src/ace/Compilation/TemplateCompiler.cs:line 40
at ACE.Program.GetTemplate(FileInfo templateFile, ILogger`1 logger, TemplateType& templateType) in /src/ace/Program.cs:line 317
at ACE.Program.Estimate(FileInfo templateFile, String scopeId, String resourceGroupName, String location, EstimateOptions options, CommandType commandType) in /src/ace/Program.cs:line 216
at ACE.Program.<>c.<b__1_15>d.MoveNext() in /src/ace/Program.cs:line 64
--- End of stack trace from previous location ---
at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<b__0>d.MoveNext()

Seems like something may be off with the package. Let me check if the published artifact contain all the dependencies.

@villanisaac-kr quick question - in the title you stated, that it's not working for OSX. I believe you have that OS and tries to run ACE from a binary and via Docker, is that correct?

@villanisaac-kr quick question - in the title you stated, that it's not working for OSX. I believe you have that OS and tries to run ACE from a binary and via Docker, is that correct?

Yes I'm using OSX. I thought that by using  the Docker image all the libraries would've been included. But just got the same missing library.

@villanisaac-kr Could you please check the latest binary for OSX from here? https://github.com/TheCloudTheory/arm-estimator/releases/tag/1.2.1
I believe the problem may have been caused by missing OSX binary for Terraform parser. Regarding Docker I need to dive deeper into that as for now I cannot reproduce it.


Dribbble ShotActivity status/nav bar fading is broken

This screen used the ChromeFader class to fade the status/nav bar if the screen is being drag dismissed downward/upward respectively. This appears to be broken and instantly makes them transparent as soon as a drag starts.
The ChromeFader class appears to be working fine elsewhere e.g. DN StoryActivity so seems specific to this screen. Perhaps the move to injecting this class?

Watch the status bar here:

I took a look at this, looks like it's happening because the injection of ChromeFader happening before the activity setContentView (because of the lazy delegate). The ChromeFader sees that the activity has no content view yet and sets status bar alpha to 0.
You could either:

inflate the activity right away (will have to make binding a lateinit var and remove delegate)
reference the binding before injecting Dagger to trigger the lazy delegate
remove the ChromeFader from Dagger

I gave a shot at the third solution here: #739


[C++] Fix clang's -Winconsistent-missing-override in proto reader/writer.

The "override" is not added on purpose to remain compatible with Protobuf < 22.x, as already written in the comment on top of these two functions.
CC @veblush as the author of this code.
Note: I am personally not super enthousiastic about this change. As an alternative, I can propose to selectively add the override keyword, based on the value of the PROTOBUF_VERSION macro (comparing it to 4022000). Tell me if you prefer this version instead.

Reassigning to @veblush who added the linter exception. I think the cleaner solution would be to exchew overrides entirely in this file, and expand the linter exception's effect. In any case, I'd recommend adding a // TODO([DATE]): remove linter exception for when we can stop supporting Protobuf < 22.x

Thank you for the PR!
I'd prefer to use PROTOBUF_VERSION rather than clang pragma. Would you make that change?

Ok, I have pushed something without testing, but your CI will test for me ;)

(note: it seems I am not allowed to add new labels (to add release notes: no), so the CI is not triggered).

Ping

Sorry for being late here. Thank you for the change and I'm going to trigger the test.

Can please someone add the [kokoro:run](https://github.com/grpc/grpc/labels/kokoro%3Arun) label to restart the CI ?

Clang-tidy still begins to complain about this change. You may want to keep this NOLINTNEXTLINE(modernize-use-override) annotation.
/root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/include/grpcpp/support/proto_buffer_reader.h:127:16: error: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override,-warnings-as-errors]
  virtual bool ReadCord(absl::Cord* cord, int count)
  ~~~~~~~~     ^
Suppressed 7670 warnings (7616 in non-user code, 53 NOLINT, 1 with check filters).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
1 warning treated as error

2023-08-29 20:03:30,078 FAILED: include/grpcpp/support/proto_buffer_reader.h [ret=1, pid=3781, time=17.4sec]
2023-08-29 20:03:30,083 WAITING: ETA 586.1 sec; 2342 queued, 16 jobs running, 301 complete, 1 failed (load 16.00) next: src/core/ext/filters/client_channel/client_channel_service_config.cc @ 1.00 cpu
2023-08-29 20:03:30,311 WAITING: ETA 585.7 sec; 2341 queued, 16 jobs running, 302 complete, 1 failed (load 16.00) next: src/core/ext/filters/client_channel/client_channel_service_config.h @ 1.00 cpu
2023-08-29 20:03:30,406 7618 warnings generated.
/root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/include/grpcpp/support/proto_buffer_writer.h:157:16: error: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override,-warnings-as-errors]
  virtual bool WriteCord(const absl::Cord& cord)
  ~~~~~~~~     ^
Suppressed 7670 warnings (7616 in non-user code, 53 NOLINT, 1 with check filters).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
1 warning treated as error

Ok, I have just updated the pull request.

Note: the CI is still not fully green, but I doubt this is linked to this pull request.


[Question] How to get the VIP-Bench?

Question
Hello @mu-cai ,
Thanks for your excellent work. How can we get the VIP-Bench?

Hi Kaining, the dataset will be released today. Thanks for your interest of our work!

Wow, this is helpful.

Hi @mu-cai , Is the dataset ready to use now?

Hi Kaining, you can use your dataset here: https://huggingface.co/datasets/mucai/ViP-Bench. Thanks!


Update game.py

@amastis we're avoiding using exit from the sys module since this is causing errors with the exe version.


Migrate to ts

migrate the whole project to ts

so no more js?

nah, the bot will still run on js, but at backed it is built with ts, it just makes my work harder to code, but get 100% assurance that there will be no runtime errors. technically you have to recompile the bot to js, which spits out js. dw, this is automated


Add additional C++ file extensions

Lately I've been working on a project that uses .cc and Discord RPC wouldn't show the right icon, so let's fix that. I've added all the widespread C++ extensions including cxx and c++ (with the corresponding header file extensions).
This should fix this problem:

I also added spacing for PowerShell file extensions.

Thanks :+1:


Update DevGuide.md to match the change in fetchDependencies.gradle

Commit a71706569d64dfa84bc184107790494c3b3bc32f changes the fidb download URLs from https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/.fidb to https://github.com/NationalSecurityAgency/ghidra-data/raw/Ghidra_${RELEASE_VERSION}/FunctionID/.fidb in gradle/support/fetchDependencies.gradle, but the manual download instructions in DevGuide.md was not updated.

Bah, I seem incapable of remembering  to update that. As a user, do you find that document useful now that the README.md has been improved?

@ryanmkurtz Yes, I find DevGuide.md still useful. I followed the manual download instructions to build Ghidra on an offline system.

Might I ask why you followed those instructions rather than just running the fetchDependencies.gradle script and moving over the dependencies directory?  I've been playing with passing an offline flag to the script that just outputs a series of linux commands like we have in the instructions.  It works nicely, but I'm wondering what the real value is since that directory can just be moved around.

Because long time ago, the dependencies were not saved in one single directory, so at that time, I downloaded them individually and copied them to the corresponding directories. It was just a personal preference. I agree nowadays it's simpler to just run the script and move the dependencies directory, so I don't object removing the manual download instructions.


[staging] coreutils: 9.1 -> 9.3 and re-enable hole seeking on Darwin

Description of changes
The changes are mostly minor; numerous bugs have been fixed and a few new command-line options have been added.
Changelog: https://lists.gnu.org/archive/html/coreutils-announce/2023-03/msg00000.html

Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Padraig said it should be better now ;)

I don't expect this is suitable for 23.05 anymore.

Any objections to merging now?

I dont have any ;)

This seems to have caused a regression in cp with older kernels with a problematic copy_file_range call. A fix is mentioned in the latest changelogs: https://github.com/coreutils/coreutils/commit/3621d51c9d7afd87c092abb1db32188ce0d1023d

Oh no, impurity in the form of build-time kernel version detection?

It's unclear, it looks like version 9.1 on my kernel handles the response from copy_file_range more gracefully instead of dying.

@tpwrules: perhaps 9.4 will help with that?  (PR #252165)

cp, mv, and install now avoid copy_file_range on linux kernels before 5.3 irrespective of which kernel version coreutils is built against, reinstating that behavior from coreutils-9.0.


UDFs are no longer executed in a thread

Describe the bug
In Datafusion 28 and below, UDFs where executed in a separate thread when writing to parquet. The example code below does not fail the assertion in version 28 but does in version 30 and git main.
If you have a UDF that expects to be running in a thread, or does some form of blocking computation then this change means your previously parallel dataframe plan becomes serial.
I couldn't spot anything in the release notes about this.
To Reproduce
deps:
[dependencies]
tokio = { version = "^1.0", features = ["rt-multi-thread", "full"] }
datafusion = { version = "=30", default-features = false, features = ["encoding__expressions", "zstd"] }

code:
#[tokio::main(flavor = "multi_thread", worker_threads = 10)]
async fn main() {
    use std::sync::Arc;
    use datafusion::arrow::array::{ArrayRef, BooleanArray};
    use datafusion::arrow::datatypes::DataType;
    use datafusion::logical_expr::Volatility;
    use datafusion::parquet::file::properties::WriterProperties;
    use datafusion::physical_plan::functions::make_scalar_function;
    use datafusion::prelude::*;

    let config = SessionConfig::default();
    let ctx = SessionContext::with_config(config);
    let df = ctx
        .read_parquet("input.parquet", ParquetReadOptions::default())
        .await
        .unwrap();

    let main_thread_id = std::thread::current().id();

    let func = create_udf(
        "some_func",
        vec![DataType::Binary],
        Arc::new(DataType::Boolean),
        Volatility::Immutable,
        make_scalar_function(move |args: &[ArrayRef]| {
            let func_thread_id = std::thread::current().id();
            assert_ne!(main_thread_id, func_thread_id);
            return Ok(Arc::new(BooleanArray::from(vec![true; args[0].len()])));
        }));

    let df = df
        .select(vec![col("hash"), func.call(vec![col("hash")])])
        .unwrap();

    let props = WriterProperties::builder()
        .build();
    df.write_parquet("some_dir/", Some(props)).await.unwrap();
}

Expected behavior
Sync UDF functions should be executed in a blocking thread pool.
Additional context
I thought this might be related to https://github.com/apache/arrow-datafusion/pull/7205, but it doesn't appear to be culprit.

hi @orf  -- I am not sure that DataFusion guarantees that udfs will be running in a separate thread or that they can do blocking operations without stopping DataFusion
What is your blocking udf doing? Is it doing network operations or something where having async udfs (like in  https://github.com/apache/arrow-datafusion/issues/6518 ) would help?


feat(op-reth): [User API enhancements] Ability to forward transactions to the sequencer

Describe the feature
Diff Category: User API enhancements | API Backend
Overview
In Optimism, the sequencer is responsible for transaction ordering and block building. As such, transactions must be forwarded to it in order to land onchain. In this case, the node acts as a proxy between the user and the sequencer.
Blocked by:

#3794

Where should this feature live?
This feature will probably need to be added to Reth's RPC API in the rpc crate.
For instance, here's where the eth_sendRawTransaction method is implemented:
https://github.com/paradigmxyz/reth/blob/90b6d006d1fc660c4883a4335e42f8e760c4adbe/crates/rpc/rpc/src/eth/api/transactions.rs#L381
Completion Criteria

[ ] forward transaction requests to the sequencer endpoint if the corresponding CLI flag is configured

Additional context
Resources

op-geth diff
op-erigon diff
Optimism Specs

We'd want to split this out more I assume, but just a first draft:
    async fn send_raw_transaction(&self, tx: Bytes) -> EthResult<H256> {
        let recovered = recover_raw_transaction(tx)?;

        let pool_transaction = <Pool::Transaction>::from_recovered_transaction(recovered);

        // forward the transaction to the optimism sequencer
        if self.network().chain_id() == 10 {
            let client = Client::new();

            let body = json![{
                "jsonrpc": "2.0",
                "method": "eth_sendRawTransaction",
                "params":[hex!(tx)],
                "id": 10,
            }];

            let req = Request::builder()
                .method(Method::POST)
                .uri("https://mainnet-sequencer.optimism.io")
                .header("Content-type", "application/json")
                .body(Body::from(body));

            let res = client.request(req).await?;
        }

        // submit the transaction to the pool with a `Local` origin
        let hash = self.pool().add_transaction(TransactionOrigin::Local, pool_transaction).await?;

        Ok(hash)
    }

This lgtm! Just as a reminder, this is a part of op-reth and won't live in this repo for now, so any changes should be opened as a PR on https://github.com/anton-rs/op-reth starting from the branch clabby/op-reth (which we use to see the entire diff so far).
Please don't be afraid to ask if you need any other info on contributing to this!

Closed by https://github.com/anton-rs/op-reth/pull/41


fix: minified umd build without prop-types

Fix #80
I had a hard time finding that the cross-env usage for the minified version was one of the problems.
cross-env NODE_ENV=production && cross-env TARGET=minify webpack is not setting NODE_ENV for webpack
cross-env NODE_ENV=production TARGET=minify webpack is the proper way to use cross-env

Just rebased on master to remove all conflicts

Sorry I'm closing this in favor of #109

No problem, nice to see rollup in use here :v:


Import wallet.dat

Export can do wallet.dat AND private keys, but import can only do private keys.
create import function for wallet.dat as well.

make sure not to simply replace current wallet.dat because people might miss the fact they will lose access to current wallet if they didn't backup.. since it's not the same as importing private keys, which just enriches current wallet.dat

how do I send zcl from a imported wallet to a standard wallet or other wallets?  When I send to main wallet address, it stuck on “signing transaction”. Then it failed and I get this error message”transaction was rejected by network rules”. Why can’t I transfer from “imported” wallet to a standard wallet?


A ical exporter for scheduled tasks would be nice

taskwarrior ships with a VTODO export PERL script.
I'm thinking of something what can export an iCal file with VEVENTS for, say, scheduled tasks, only.
It should be simple with the vobject library and this taskwarrior python bindinds...

Thanks for the suggestion! I think that kind of thing is a little out of scope for this project, though as you say it should be fairly easy to build using tasklib as a base. Closing this issue, but let us know if you end up writing something which we can link to in the docs!


Simplify derive

I don't think we need the derive at all, but nonetheless, this code insults my sense of beauty. Let's fix it for the sake of history.

Moving away from the derive.


Transition frontier debug assert failure -- local state desynced

Edit by @yourbuddyconner:
Related to this crash:
(monitor.ml.Error\n  (Failure \"local state desynced after applying diffs to full frontier\")\n  (\"Raised at file \\\"stdlib.ml\\\", line 33, characters 17-33\"\n    \"Called from file \\\"src/lib/debug_assert/debug_assert.ml\\\" (inlined), line 1, characters 25-33\"\n    \"Called from file \\\"src/lib/transition_frontier/full_frontier/full_frontier.ml\\\", line 606, characters 4-1023\"\n    \"Called from file \\\"src/lib/transition_frontier/transition_frontier.ml\\\", line 292, characters 4-93\"\n    \"Called from file \\\"src/lib/transition_handler/processor.ml\\\", line 57, characters 10-68\"\n    \"Called from file \\\"src/lib/rose_tree/rose_tree.ml\\\", line 79, characters 18-24\"\n    \"Called from file \\\"src/deferred_or_error.ml\\\", line 101, characters 23-33\"\n    \"Called from file \\\"src/deferred0.ml\\\", line 56, characters 64-69\"\n    \"Called from file \\\"src/job_queue.ml\\\" (inlined), line 131, characters 2-5\"\n    \"Called from file \\\"src/job_queue.ml\\\", line 171, characters 6-47\"\n    \"Caught by monitor coda\"))

coda_crash_report_2020-06-10_00-40-20.819931.tar.gz

Crashed a few minutes after start with ```
2020-06-10 01:27:26 UTC [Error] Unconsumed item in cache: $cache
cache: "cached item was not consumed (cache name = "Transition_handler__Unprocessed_transition_cache")"
[coda_crash_report_2020-06-10_01-33-18.973349.tar.gz](https://github.com/CodaProtocol/coda/files/4755679/coda_crash_report_2020-06-10_01-33-18.973349.tar.gz)

2020-06-10 01:18:55 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
        exn: "(monitor.ml.Error\n  (Failure \"local state desynced after applying diffs to full frontier\")\n  (\"Raised at file \\\"stdlib.ml\\\", line 33, characters 17-33\"\n    \"Called from file \\\"src/lib/debug_assert/debug_assert.ml\\\" (inlined), line 1, characters 25-33\"\n    \"Called from file \\\"src/lib/transition_frontier/full_frontier/full_frontier.ml\\\", line 606, characters 4-1023\"\n    \"Called from file \\\"src/lib/transition_frontier/transition_frontier.ml\\\", line 292, characters 4-93\"\n    \"Called from file \\\"src/lib/transition_handler/processor.ml\\\", line 57, characters 10-68\"\n    \"Called from file \\\"src/lib/rose_tree/rose_tree.ml\\\", line 79, characters 18-24\"\n    \"Called from file \\\"src/deferred_or_error.ml\\\", line 101, characters 23-33\"\n    \"Called from file \\\"src/deferred0.ml\\\", line 56, characters 64-69\"\n    \"Called from file \\\"src/job_queue.ml\\\" (inlined), line 131, characters 2-5\"\n    \"Called from file \\\"src/job_queue.ml\\\", line 171, characters 6-47\"\n    \"Caught by monitor coda\"))"
2020-06-10 01:18:55 UTC [Error] Unconsumed item in cache: $cache
        cache: "cached item was not consumed (cache name = \"Transition_handler__Unprocessed_transition_cache\")"

2020-06-10 01:43:56 UTC [Error] Unconsumed item in cache: $cache
coda_crash_report_2020-06-10_01-43-55.125789.tar.gz

coda_crash_report_2020-06-10_01-35-53.525361.tar.gz

2020-06-10 01:49:36 UTC [Info] Coda daemon is now doing ledger catchup
2020-06-10 01:49:36 UTC [Info] Sending transition_chain_proof to peer with IP <IP_ADDRESS>
2020-06-10 01:50:06 UTC [Info] failed to reset stream (this means it was probably closed successfully): $error
error: "RPC #83 failed: "internal RPC error error: unknown stream_idx""
2020-06-10 01:51:13 UTC [Fatal] after lock transition, the best tip consensus state is out of sync with the local state -- bug in either required_local_state_sync or frontier_root_transition.
2020-06-10 01:51:13 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
exn: "(monitor.ml.Error\n  (Failure "local state desynced after applying diffs to full frontier")\n  ("Raised at file \"stdlib.ml\", line 33, characters 17-33"\n    "Called from file \"src/lib/debug_assert/debug_assert.ml\" (inlined), line 1, characters 25-33"\n    "Called from file \"src/lib/transition_frontier/full_frontier/full_frontier.ml\", line 606, characters 4-1023"\n    "Called from file \"src/lib/transition_frontier/transition_frontier.ml\", line 292, characters 4-93"\n    "Called from file \"src/lib/transition_handler/processor.ml\", line 57, characters 10-68"\n    "Called from file \"src/lib/rose_tree/rose_tree.ml\", line 79, characters 18-24"\n    "Called from file \"src/deferred_or_error.ml\", line 101, characters 23-33"\n    "Called from file \"src/deferred0.ml\", line 56, characters 64-69"\n    "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5"\n    "Called from file \"src/job_queue.ml\", line 171, characters 6-47"\n    "Caught by monitor coda"))"
2020-06-10 01:51:14 UTC [Error] Unconsumed item in cache: $cache
cache: "cached item was not consumed (cache name = "Transition_handler__Unprocessed_transition_cache")"
2020-06-10 01:51:14 UTC [Error] Unconsumed item in cache: $cache
cache: "cached item was not consumed (cache name = "Transition_handler__Unprocessed_transition_cache")"
2020-06-10 01:51:14 UTC [Error] Unconsumed item in cache: $cache
cache: "cached item was not consumed (cache name = "Transition_handler__Unprocessed_transition_cache")"
2020-06-10 01:51:14 UTC [Error] Unconsumed item in cache: $cache
cache: "cached item was not consumed (cache name = "Transition_handler__Unprocessed_transition_cache")"
coda_crash_report_2020-06-10_01-51-14.647416.tar.gz

coda_crash_report_2020-06-10_01-55-08.816213.tar.gz

coda_crash_report_2020-06-10_02-08-09.780844.tar.gz

Same issue with Unconsumed item in cache:
coda_crash_report_2020-06-10_01-45-54.705615.tar.gz

2020-06-10 02:05:16 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
exn: "(monitor.ml.Error\n  (Failure "local state desynced after applying diffs to full frontier")\n  ("Raised at file \"stdlib.ml\", line 33, characters 17-33"\n    "Called from file \"src/lib/debug_assert/debug_assert.ml\" (inlined), line 1, characters 25-33"\n    "Called from file \"src/lib/transition_frontier/full_frontier/full_frontier.ml\", line 606, characters 4-1023"\n    "Called from file \"src/lib/transition_frontier/transition_frontier.ml\", line 292, characters 4-93"\n    "Called from file \"src/lib/transition_handler/processor.ml\", line 57, characters 10-68"\n    "Called from file \"src/lib/rose_tree/rose_tree.ml\", line 79, characters 18-24"\n    "Called from file \"src/deferred_or_error.ml\", line 101, characters 23-33"\n    "Called from file \"src/deferred0.ml\", line 56, characters 64-69"\n    "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5"\n    "Called from file \"src/job_queue.ml\", line 171, characters 6-47"\n    "Caught by monitor coda"))"
2020-06-10 02:05:16 UTC [Error] Unconsumed item in cache: $cache
cache: "cached item was not consumed (cache name = "Transition_handler__Unprocessed_transition_cache")"

Again on stage Catchup
2020-06-10 02:18:38 UTC [Info] Coda daemon is now doing ledger catchup
2020-06-10 02:19:13 UTC [Fatal] after lock transition, the best tip consensus state is out of sync with the local state -- bug in either required_local_state_sync or frontier_root_transition.
2020-06-10 02:19:13 UTC [Fatal] Unhandled top-level exception: $exn
coda_crash_report_2020-06-10_02-19-14.177339.tar.gz

same

Related to this crash:
UTC [Error] Unconsumed item in cache: $cache
cache: "cached item was not consumed (cache name =
coda_crash_report_2020-06-10_02-23-18.455018.tar.gz
"Transition_handler__Unprocessed_transition_cache")"

coda_crash_report_2020-06-10_02-40-30.246996.tar.gz

Same issue, this happened to a lot of member
Even the error log is the same

.coda_config1, .coda_config2
too big to upload directly.
refer below.
https://drive.google.com/file/d/13SeWGqihQT43Y_Qfdvj5tLiCV1qFGfoL/view?usp=sharing

Had the same problem:
coda_crash_report_2020-06-10_03-10-03.546959.zip

inv0ke -

I think the crash is related with "root" account on Ubuntu.
With root account, the following warning shows up when installing:
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start

Ver Cluster Port Status Owner    Data directory              Log file
10  main    5432 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
update-alternatives: using /usr/share/postgresql/10/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
Setting up postgresql (10+190ubuntu0.1) ...
Setting up coda-testnet-postake-medium-curves (0.0.12-beta+406048-feature-bump-genesis-timestamp-3e9b174-PV48525e92) ...
Processing triggers for systemd (237-3ubuntu10.38) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...

If switching to normal user, the node can be started smoothly :)

This issue just popped up again and can be reliably reproduced by different testnet participants when attempting to sync a new node https://discordapp.com/channels/484437221055922177/583400552487059546/720773819437940758

I scanned through the log and extracted the transition frontier visualization at the point that the local state sync invariant broke (after applying the diffs that broke it). The visualization doesn't make sense. Seems to indicate to me that, either, the frontier is very malformed, or the visualization code is really messed up. Visualization code hasn't been touched in a while, so it's worth spot checking it, but it should be pretty straight forward. Not sure how the frontier could ever look like this.

Crashed a few minutes after start with  2020-06-10 01:27:26 UTC [Error] Unconsumed item in cache: $cache cache: "cached item was not consumed (cache name = \"Transition_handler__Unprocessed_transition_cache\")"
coda_crash_report_2020-06-10_01-33-18.973349.tar.gz

The unconsumed item in cache errors are non-fatal, and they actually are triggered by the root cause of this issue, the required_local_state_sync assertion failing. These errors often trigger on a graceful crash since they are essential just saying "you didn't finish using this object yet", which is true since we are crashing.

If switching to normal user, the node can be started smoothly :)

I believe this problem is temporal in nature. By that, I mean, this bug can trigger depending on when you join the network. If that's the case, it may have started working for you due to time passing, and not by switching users. If you see this bug again, please let me know. In the meantime, we will continue investigating.

Closing this issue as it is stale for a long time.


missing clarification about synchronization

what if WdfRequestForwardToIoQueue and WdfIoQueueRetrieveNextRequest gets called from two different threads for same queue object ?
Adding this details in document will give more clarity to developer about synchronizing manualDispatch Queue.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 16abdc96-df03-59fa-b911-b8603615043b
Version Independent ID: 0e73ceb4-0071-b278-e3eb-d6f5e0019622
Content: WdfIoQueueRetrieveNextRequest function
Content Source: wdk-ddi-src/content/wdfio/nf-wdfio-wdfioqueueretrievenextrequest.md
Product: windows-hardware
GitHub Login: @windows-driver-content
Microsoft Alias: windowsdriverdev

Interesting question, @pjprajap. Thanks for asking.
@tedhudek, can you take a look and prioritize this appropriately?

Hi @pjprajap, not sure I follow the question.  If thread 1 is forwarding a request and thread 2 is retrieving it, what is ambiguous?

Closing due to lack of activity, please re-open if needed


A throttled AsyncSequence doesn't yield its parent sequence's final value

Not sure if this is expected behavior.
I make an AsyncStream called stream, and a throttled sequence from that stream throttledStream. I yield values into stream for a while, and eventually call finish() on its continuation. All the while in the background I am iterating through throttledStream - when finish() gets called on stream, my iteration through throttledStream finishes, but I don't get the final value that was sent to stream.
Sorry this is kind of a complicated way to explain the issue, but I will make a reproducible example project if this behavior sounds like it is indeed not intended!

I should add, I guess it's the "penultimate" value I'm looking for - I realize that "nil" is the final value in a "finished" AsyncStream

Is it a throttle with “latest” ? If not then it can be normal that the penultimate element is skipped.

Is it a throttle with “latest” ? If not then it can be normal that the penultimate element is skipped.

Yes, latest = true, just calling let throttledStream = stream.throttle(for: .seconds(1), latest: true), for example.

@gregcotten
Keep in mind that AsyncSequence is a pull mechanism whereas Combine is a push mechanism for instance. It means that the throttling is applied to the pace of the consumer, not the producer.
in your case, let’s say you have a throttling duration of 1s, here is what I suspect is happening at the end of the timeline:
|
| x -> last received element by the consumer, a new throttling start time is set, ending in 1s
| y -> new element is discarded because it is too close to ‘x’ (the time elapsed since the start is < 1s)
|
| z -> last element is discarded because it is too close to ‘x’ (the time elapsed since the start is < 1s)
|
| nil -> the upstream sequence is finished before the theoretical end of this throttling session
|
| ——> theoretical end for the current throttling session (start + 1s)
V

In the end the consumer doesn’t receive the penultimate element but the last that matches un throttling session’s end, that is to say ‘x’.
I hope this helps. @FranzBusch could also chime in the confirm or not.

Yes, this behavior makes sense to me, but I think it might be nice to find a way to always get the penultimate value somehow... Would a PR be welcome for this?

Yeah I can see how that would be useful. Let’s say you throttle a text field (like a search field), with the current implementation you would not have the last character … which would be wrong search wise.
@FranzBusch what’s your take on that ? Should we switch to another implementation that:

iterate over the upstream sequence in a dedicated Task and push the values into a “Throttle” state machine
run a “tick” periodically in a dedicated task and push each “tick” into the “Throttle” state machine
for every “tick” use the “Throttle” state machine to retrieve the appropriate value and push it into a kind of “unicast buffered state machine”
expose the “unicast buffered state machine” to the consumers

It would be much closer to what throttling does in push based system … but with the drawback of not be based on the consumer pace.

Couldn't we just change the implementation of throttle to check if there is a latest value when the upstream returns nil and then return the latest value. IMO we could also make this the default but would like to get @phausler  on this as well.

That makes sense to me. Would the final value still be throttled or debounced at the specified interval, or would we break the contract for the "final" value?

Good question. I would like to understand what the current standard in other frameworks is. If we could get a little comparison going here that would be nice to make an informed choice.

@FranzBusch My initial thought would be to respect the specified throttle interval when deciding when to emit the final value.

Combine for example used the rule for throttle and debounce: that the iterative output (not the signals to determine failure or finishedness) are the rate limited things. So specifically for throttle; it means that even the last value should not emit any faster than the requested throttling rate.
The intent for folks using throttle is that they don't ever want a rate beyond a given value. Making the last value ignore that would break folks expectations.

It might be difficult to achieve with the current implementation since we have no way to determine that a value is the final value. We cannot postpone the emitting until the throttling duration is finished. That's why I suggested to rely on a tick mechanism but it makes things significantly more complex and changes the way to upstream sequence is being iterated over.

Thanks for clarifying the behaviour @phausler. I think that makes sense but it also means we will be dropping the final values in some cases which IMO is correct. If you throttle you have to live with the fact that you might be losing values which you are already due to the rate limiting.

@FranzBusch that wasn't how I interpreted @phausler's comment. I took it to mean we should always emit the final value, but only after the specified throttle interval has elapsed.

@FranzBusch it means that if you throttle a TextFields you might loose the last value, which will be an issue when performing a search for instance.

no, I meant that the value is not expected to be emitted earlier than the interval, termination on the other hand is not a participant of that emission so therefore it can happen before the interval.
Per text fields; I think (just like other button presses) debounce is usually the right tool. Sadly we don't have any capacitors handy ;)

My issue is that I want to update the progress of a long running task, and would like to not miss the final "I'm at 100% complete" progress update ever, which seems like a fair request.

Per text fields; I think (just like other button presses) debounce is usually the right tool. Sadly we don't have any capacitors handy ;)

yep sure. I was just trying to illustrate my point with a real-life use case but I still think we should not be able to skip the last element since some use cases will rely on that.

no, I meant that the value is not expected to be emitted earlier than the interval, termination on the other hand is not a participant of that emission so therefore it can happen before the interval.
Per text fields; I think (just like other button presses) debounce is usually the right tool. Sadly we don't have any capacitors handy ;)

I'm fairly certain the current debounce implementation will not emit the final value in all cases.

Re: textfield search, not all cases you want to debounce. Throttling can be acceptable in certain use cases.

to be clear; it is reasonable to have opt-in behaviors or overloads of algorithms that adjust these fine points. I was just commenting on the behavior implemented for other systems.

We can definitely have a latest: true option here. The implementation should also be quite trivial, if we get nil from the upstream and we have a last value and latest = true then we can do a Task.sleep until the timeout has passed.

I'm fairly certain the current debounce implementation will not emit the final value in all cases.
If that happens and you can reproduce it please open a separate issue here.

@phausler just put up a PR https://github.com/apple/swift-async-algorithms/pull/292 to change the behaviour. We would appreciate if you all could check that it is doing what you expected it to do now.

@phausler just put up a PR #292 to change the behaviour. We would appreciate if you all could check that it is doing what you expected it to do now.

Yes, this does exactly what I expect now!


[Bug]: The save button is missing for authenticated API in import modal

Is there an existing issue for this?

[X] I have searched the existing issues

Description
The save button is missing for authenticated API in import modal

Steps To Reproduce

Import an application that is connected to authenticated API
Now Add all the details required for the data source authentication
The save button i snot displayed in the modal

Public Sample App
No response
Version
release

Raising severity as this bug prevents user from importing authenticated apis

now merging resolution for this bug

Fixed


load DR markets for currently logged in user

closes https://github.com/AugurProject/augur/issues/1186
loads all currently logged in user's markets if they are designated reporter.

tested, worked perfectly 👍


How to apply countDistinct to DataFrameGroupBy

Is there any way to apply countDistinct to DataFrameGroupBy like the pyspark code below? I've been trying to find a workaround but no luck.
# pyspark example 1
sdf.groupBy('x').agg(F.countDistinct('y'))

# pyspark example 2
sdf.groupBy('x').agg(F.expr('count(distinct(y))'))

Out of curiosity, is there such way in Pandas?

In Pandas, you can do:
df.groupby('x').agg({'y': 'nunique'})

or
df.groupby('x').agg({'y': lambda x: x.nunique()})

https://stackoverflow.com/questions/18554920/pandas-aggregate-count-distinct

See https://github.com/databricks/koalas/pull/512

Cool. Thanks!


Improve documentation

Improves some things that are not documented clearly enough that @christopherbiscardi discovered while using the template.
Resolves #171

smol suggestion. Reasoning being the same as others in a similar vein. otherwise 👍🏼 lgtm


[release-1.5] Fix an array pointer overflown problem in Operator

This is an automated cherry-pick of #20750

/retest


ActivatedTest.PageControlTiming fails randomly

FAILED:  AllTests.Fuse.Navigation.Test.ActivatedTest.PageControlTiming
         Assertion failed in 'PageControlTiming', at C:\Users\erik\src\fuselibs\Source\Fuse.Navigation\Tests\Activated.Test.uno:165
           Expected: 1
           But got : 0

I'm waiting on my other PRs to test this. I changed something about how this works, and fixed a related defect.

I can't get this to happen locally on the improved PageControl branch. Doesn't mean it's gone though. :/

This hasn't been reported in a while, so I assume it's stable now.


Destructuring assignments are producing errors with noImplicitAny

I found an issue in the current nightly builds relating to destructuring assignments with noImplicitAny enabled. The actual code was more complicated, but it can be simplified to the following:
let [a, b, c] = [1, 2, 3];

The variables appear to be typed correctly on hover, however, 3 errors are produced inside the assignment stating that a, b, and c have an implicit type of any.
It can be silenced by the following, but that's not ideal:
let [a, b, c] = <[number, number, number]>[1, 2, 3];

TypeScript Version:
Searching backwards, it first appears in:

nightly (1.9.0-dev.20160409)

Expected behavior:
There shouldn't be an implicit any error in this case.
Actual behavior:
An erroneous error is produced.

looks like a regression from https://github.com/Microsoft/TypeScript/issues/6935

@sheetalkamat Any update on this issue? I came across the problem again when trying to build my code-base with the 2.0 beta.

This bug is going to be encountered fairly frequently using TS 2.0 beta


Handle empty paths in appendPath

Fixes #17

Thanks @lordjeb! wasn't sure if you'd consider this a bug or not.


kube-state-metrics provides wrong memory metrics for pods

What happened?
I installed latest version of kube-prometheus. with kube-state-metrics 1.9.7 on my self managed kubernetes cluster
Did you expect to see some different?
I expected that the metric service will provide metrics about pod memory consumption.
I verified the metrics with:
$ kubectl top pod my-app
and on the corresponding worker node with:
$ docker stats
It shows that the metric data concerning memory usage of a pod shows the double size as it should be.
How to reproduce it (as minimally and precisely as possible):
You can compare the metric data with kubectl top and docker stats
After I uninstalled kube-prometheus stack and installed instead the metric-server all memory is displayed correctly and kubernetes scheduling was again as expected
Does anybody know how this can happen and what I can do about this issue?
See also discussions here:
https://stackoverflow.com/questions/64582065/why-is-openjdk-docker-container-ignoring-memory-limits-in-kubernetes
https://stackoverflow.com/questions/64440319/why-java-container-in-kubernetes-takes-more-memory-as-limits
Environment
Debian Buster
Kubernetes 1.19.3
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Manifests:

kube-state-metrics 1.9.7

You are most likely scraping both the /metrics/cadvisor as well as the /metrics/resource endpoints, which causes this. At least we've had many reports of this on slack lately. How did you install kube-prometheus and with what configuration?

I cloned the kube-prometheus repo from github and than I did:
$ kubectl create -f manifests/setup
$ until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
$ kubectl create -f manifests/

I did use kustomize only to add an ingress setup and add some Grafana environment variables to support SMTP.
So I think from the core installation of kube-prometheus I did follow the given instructions.
The thing which I did not understand is that  kubectl top pod ...  returned wrong metrics. After I uninstalled kube-prometheus and I installed the 'metrics-server' kubectl top shows the correct data. And than also my kubernetes cluster behaves as expected. The strange thing was, that when I looked into the memory consumption of my pods (I am running only java services) the numbers were double in the size. But I did not realize that these numbers where wrong. I also used k9s to monitor my cluster.
So the most of my time I invested to optimize my java containers. After uninstalling kube-prometheus and installing the metrics-server I can see now that everything is fine with my pods and also with my cluster.

Sorry for taking so long to reply, but could you elaborate in what way they seem "wrong"? Also which version of kube-prometheus are you installing?

I checked out the master branch from github. This was a month ago.
The result was that memory usage was measured  in double, so that I had to high volumes leading to a wrong behaviour. I uninstalled everything and installed the metrics-server to get the correct data.

Would you be open to trying again and sharing some data with us so we could debug the situation?

I'm sorry, I can't test this again at the moment because my environment can't be changed now.

I fear that we must close this issue because I can not provide more information or do a new testing. Do you have any suspicions about what might have been the root of my problem?

No problem! Please feel free to open a new issue should you encounter anything in the future! :)

I am having this issue with the double size of the memory with

kube-prometheus-stack
appVersion: 0.49.0
version: 17.2.2

I can reproduce this with a container running a single process with about 7GB memory:
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     **17 root      20   0 7720256   7.4g   1740 S   6.2   5.9   0:32.41 tail**
      1 root      20   0    4260    592    528 S   0.0   0.0   0:00.05 sleep
      6 root      20   0    5992   3980   3392 S   0.0   0.0   0:00.02 bash
     15 root      20   0    5732   3508   3256 S   0.0   0.0   0:00.00 bash
     18 root      20   0    5992   3960   3372 S   0.0   0.0   0:00.02 bash
     27 root      20   0    7860   3616   3052 R   0.0   0.0   0:00.00 top

The grafana dashboard shows me that there are about 14 GB memory used.

When I use TICK stack with the docker input plutin, the usage of about 7GB can be seen. Also have a look at the host memory, which increased for 7 GB only.

In prometheus/kube-state-metrics, you can also reproduce this by having a process that uses more than the half of the memory allowed by kubernetes limit:
For example:

k8 memory limit is on 27 GB
script in container uses 15 GB

result:
via TICK stack:

via kube-prometheus-stack:

This is impossible...
Information about the kubernetes Node which is running this:
zwei     Ready                      <none>                 216d   v1.19.7    <IP_ADDRESS>   <none>        Ubuntu 20.04.1 LTS   5.4.0-62-generic   docker://19.3.14

/reopen

Ran into the same issue. I believe this is because one tool measures the total memory usage while another only measures the working set.

I'm facing same issue.
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2
If anyone has resolved it, kindly let me know .

facing same issue
already changes container_memory_working_set_bytes with cotnainer_memory_usage_bytes in kube-prometheus-stack/templates/grafana/dashboard-1.14/k8s-resources-pod.yaml
but in grafana still show twice the true values of pod resoources from kubectl top pod or from kubernetes dashboard


Improved compatibility, new configs, and balance tweaks

Summary
Hey, awesome mod! I have been using it and made some tweaks along the way, most of them balance related.
Questions and feedback welcome!
(Below is a copy paste from the Changelog entry in the README)
Fixes

No longer despawns all item drops if Artifact of Command is disabled.
Mod will disable itself during Bulwark's Ambry to avoid a bug causing Artifacts to not be rewarded on successful clears.
[DropInMultiplayer] Focusing (alt-tab) the Application will fix loot being non-interactable if a player joins after a level has loaded. This issue will resolve automatically at the beginning of the next stage if no action is taken.
Resolve NRE log spam from F3 keybind when no items can be pulled.

Configuration File Options

Base drop rate
Disable player-based drop rates
Disable swarm drop rate modifier
Minimum drop rate
Drop rate mulitplier

Dynamic Drop Rates
There are a lot of details here, so it can be hard to understand how the game actually feels with these changes.
Here is what we aimed for:

Roughly 1 item per minute (this inevitably speeds up later in the game).
More consistent experiences with varying group sizes. (Roughly the same item rate as solo play.)
More consistent experiences with and without Artifact of Swarms.

Balance Changes

Compensate drop rate for Artifact of Swarms (the game doubles spawns, but reduces drop rates by as little as 15%).
Ensures drop rates are appropriate for the number of players, since all drops are shared now.
Account for additional factors used by the game when calculating drop rates.

Artifact of Swarms lowers drop chances. (Lemurian w/ Swarms: 5%) vs (Lemurian w/o Swarms: 7.9%)
Elites have higher drop rates than normal enemies. (Normal Lemurian: 7.9%) vs (Elite Lemurian: 19%)
Some enemies have a higher base drop rate than others. (Normal Beetle: 5%) vs (Normal Scavenger: 43.2%)
Some entities have a native drop rate of 0%. (Mending healing orbs, TheBackup drones, Void Infestors)

Bad Luck Protection: A 4 player lobby will have a base drop rate of 1.25, so it is possible for drops to feel scarce. We prevent unlucky streaks by adding all drop chances to a pool, and increasing drop rates after it reaches 100%. Bad luck protection only kicks in if the survivors item count is falling behind (fewer item drops than expected or low item drops per minute).

Example: Survivors have killed 80 enemies with a drop rate of 1.25%, adding up to 100%.

next enemy has a 1.25% drop rate, so the drop rate will be bumped to 2.5%. (cumulative chance = 101.25%)
next enemy has a 3% drop rate, so the drop rate will be bumped to 7.25%. (cumulative chance = 104.25%)
next enemy has a 1.25% drop rate, so the drop rate will be bumped to 6.75%. (cumulative chance = 105.5%)
next enemy has a 5% drop rate, so the drop rate will be bumped to 10.5%. (cumulative chance = 110.5%)

Last two pushes fixed a typo in the drop rate report and an issue where drops where being added to total items twice.


TN-1163 removed u in dict_tools.py

Hello @John0Randolph! Thanks for submitting the PR.

In the file portal/dict_tools.py, following are the PEP8 issues :

Line 43:39: E231 missing whitespace after ':'
Line 46:41: E231 missing whitespace after ':'


Asset reveiew

Please review the following token assets:
📑 Description

Token Name: NGNC Coin
Project Website: https://www.linkio.world/ngnc
Explorer URI: https://base.blockscout.com/address/0xA971B44246a55b3676ab2E66a4732Fa4cce2EA13

✅ Checks

[ ✅ ] I created a new folder with the token address (all in lowercase for EVM chains, case sensitive for Solana)
[ ✅] I added the token's logo as a 32x32 PNG file, named logo-32.png
[ ✅] I added the token's logo as a 128x128 PNG file, named logo-128.png
[ ✅] I added the token's logo as a SVG file, named logo.svg
[ ✅] My SVG logo is a proper SVG and not some base64 stuff
[ ✅] My documentation/website clearly display the token address somewhere

folder naming updaetd


Cannot identify qtile window manager

Description
Neofetch displays LGD3, I don't know what that is. I'm using qtile. Same issue with dwm

[x] Does this issue still occur in the master branch? (Required if issue)

Neofetch version
Latest as of 16 March
Screenshot

Config file
None

Reopen with logs


Change How Do I See It default link text

Problem Description:
If no identified_by data is provided the default link text is "View this item on the collection website" which is not accurate as not all links will be collection links.
Expected Behavior/Solution:
The default text should be the link itself.
Requirements:

[x] Change the default text in the HowDoISeeIt component
[x] Ensure text wrapping of link so that it does not spill out of it's container

Needed for promotion:
If an item on the list is not needed, it should be crossed off but not removed.

[x] Wireframe/Mockup - Kam
[ ] ~Committee discussions - Sarah~
[ ] ~Feasibility/Team discussion - Sarah~
[ ] ~Backend requirements - TBD~
[x] Frontend requirements - Kam
[ ] ~Are new regression tests required for QA - Amy~
[ ] ~Questions~

UAT/LUX Examples:

DEV: https://lux-front-dev.collections.yale.edu/view/text/fada5383-ef3c-4b34-98d4-4ef55a51c0b0
PRD: https://lux.collections.yale.edu/view/text/fada5383-ef3c-4b34-98d4-4ef55a51c0b0

Dependencies/Blocks:
N/A
Related Github Issues:
N/A
Related links:

Bugherd: https://www.bugherd.com/projects/284041/tasks/2275

Wireframe/Mockup:
Current:

Expected:

Approved by UAT

Closing as this ticket was marked as Done the week of 6/3.