Using s3fs with no 'LastModified' key info

I have a Ceph based Object store with S3, and I am trying to use s3fs.
The metadata of the objects do not seem to have 'LastModified' entry, which was identified using Boto3's s3.Client.head_object().
Here's a sample metadata response of one of the objects
{'ResponseMetadata': {'RequestId': 'tx0000000000000067d7a8a-005c3dce53-489594c6-in-chennai', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'openresty/<IP_ADDRESS>', 'date': 'Tue, 15 Jan 2019 12:13:07 GMT', 'content-type': 'application/octet-stream', 'accept-ranges': 'bytes', 'etag': '"ed076287532e86365e841e92bfc50d8c"', 'content-length': '12', 'x-amz-request-id': 'tx0000000000000067d7a8a-005c3dce53-489594c6-in-chennai', 'expires': 'Thu, 31 Dec 2037 23:55:55 GMT', 'cache-control': 'max-age=315360000', 'x-e-id': '<IP_ADDRESS>', 'x-elb-id': '<IP_ADDRESS>', 'x-elb-app-server': '<IP_ADDRESS>'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'ContentLength': 12, 'ETag': '"ed076287532e86365e841e92bfc50d8c"', 'CacheControl': 'max-age=315360000', 'ContentType': 'application/octet-stream', 'Expires': datetime.datetime(2037, 12, 31, 23, 55, 55, tzinfo=tzutc()), 'Metadata': {}}

When I try to use s3fs operations like open() [To open a file for reading and writing], it gives the following error
Traceback (most recent call last):
  File "/Users/aman.garg/Downloads/test/s3fs-touch.py", line 11, in <module>
    with fs.open('amantestnotebook/notebook/write_file.txt', 'rb') as f:
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 315, in open
    s3_additional_kwargs=kw)
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 1102, in __init__
    info = self.info()
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 1120, in info
    refresh=refresh, **kwargs)
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 447, in info
    'LastModified': out['LastModified'],
KeyError: 'LastModified'

Operations like ls(), mkdir(),isfile(), isdir() are working as expected.
Is there a way to use S3FileSystem without the 'LastModified' metadata? What exactly is this LastModified entry used for?

I see that this issue is very old, however this problem has been plaguing me in the context of testing with moto. I genuinely do not know if it's an issue where the mocked objects don't have LastModified attribute or not however I think a simple fix here would be to replace out['LastModified'] with out.get('LastModified', ...). I don't know what a reasonable default is, maybe None would be fine.
Would the fsspec maintainers be open to a one-liner PR to implement this change?


fix: gap between navbar-search-active and navbar

Fixes #4583
Describe the changes you have made in this PR -
I changed the the top position of navbar-search-active
Screenshots of the changes (If any) -
BEFORE

AFTER

Note: Please check Allow edits from maintainers. if you would like us to assist in the PR.

pls review the PR

pls review the PR

Doesn't the current behaviour looks better than the updated one (PR).
This update introducing a band between those two portion.
WDYT ?

pls review the PR

Doesn't the current behaviour looks better than the updated one (PR). This update introducing a band between those two portion.
WDYT ?

What do you mean by adding band

pls review the PR

Doesn't the current behaviour looks better than the updated one (PR). This update introducing a band between those two portion.
WDYT?
but it is showing in mobile view not in desktop
we may do that in desktop view also

@tanmoysrt actually it is due to absence of media query in mobile view so we can that


5.4 Kernel Upgrade for HFCL ipq60xx targets

This PR covers 5.4 upgrade for HFCL targets hfcl_ion4xe, hfcl_ion4xi, hfcl_ion4x, hfcl_ion4x_2

@esha-aggarwal18 changing the ion4xi profile image path as name incorrectly mentioned!

hfcl_ion4x has an incorrect package requirement, it should use ipq60xx not ipq807x in the profile


Doubts about Basic Auth

How do I pass user and password to do a Basic Auth Kong when connecting using gkong.NewClient(...) and make a client.CACertificates.Create(....) for example. ?

thanks man!


Dirty fix mkchain for release

Ideally we want to remove docker completely from mkchain
closes #71
closes #66

Now each time I run mkchain, it requires me to be authenticated to docker hub and prints a lot of lines of output. I think it's better to check whether the image is already pulled, and pull if it isn't.

Now each time I run mkchain, it requires me to be authenticated to docker hub and prints a lot of lines of output. I think it's better to check whether the image is already pulled, and pull if it isn't.


Running Qwen

I tried running Qwen with Langchain but didn't get any output. It is stuck.
Has anyone else got stuck at the same place?

There are several different issues with Qwen already:

https://github.com/ollama/ollama/issues/2405
https://github.com/ollama/ollama/issues/2385
https://github.com/ollama/ollama/issues/2379

Unfortunately, I have seen any solutions or workarounds yet 😞

Same here, I have been trying different models of Qwen but none of them worked for me.
@svilupp Have to tried any other way to run Qwen that would have worked for you like hugging face or any other framework?

No, gave up. I'm waiting for the GGUF to be re-uploaded.
In general, I'm having so many issues with Ollama this week, that I'll need to explore other alternatives 😢

Downloaded today latest version of ollama ..and Qwen models , it worked for me on macOS - Intel MacBookPro.

I tried running Qwen on Ubuntu 22.04 after a fresh installation of Ollama, working for me :+1:

i downloaded the latest release and tried to run qwen on ubuntu 20.. still no luck, here. and requires restart ollama service...

ok, working. done

Great 😃

It's seem good news for run Qwen with ollama.
I'll check again.

The fixed GGUF haven’t been re-uploaded yet. So you would have to download them from elsewhere (the ones in Ollama library are broken).

I tried running Qwen on Ubuntu 22.04 after a fresh installation of Ollama, working for me 👍

Great to hear!
I'm closing this issue. Please reopen if you have further issues.


[FLINK-27043][table] Removing old csv format references

This PR is preliminary work to get https://issues.apache.org/jira/browse/FLINK-26952 solved (which is blocked currently by an issue of the new csv format).
Changelog:

Remove legacy tests and mocks either using the csv format, or somehow depending on it
Added a new test extension to inject the table environment with junit 5
Rewrite TableEnvironmentITCase, TableEnvironmentTest and TableITCase with assertj/junit 5
Removed usage of old csv format from flink-batch-sql e2e test

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure


⚠ Processors has degraded performance

In c9346fe, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 49 ms

Resolved: Processors performance has improved in 79cec2c.


Make README badges inline

This makes the badges in the README inline which pushes some of the more informative content up the page.
Super small, nit change, but a bunch of other projects do the same :P

https://github.com/pypa/pip
https://github.com/requests/requests

They used to be inline (when it was restructuredText), but Kenneth changed it when converting to Markdown. I’m going to assume it was not intentional. Thanks!


Missing 2008 R2 in system requirements notes

During the doc update for Windows Server Downlevel client, it seems that the legacy Windows Server documentation have been removed and that 2008 R2 installation has merge into the legacy Windows client. Thus the notes regarding System Requirements only list Windows 7 as applicable of for Feb 2018 CU & .Net 4.5 while this is also required for 2008 R2.
I've also added a note regarding installation of .NET 4.5 that might require a reboot as this is important to know on Windows Server

@JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks


update docs and comments

When I read the docs about containerd, I found that some nit via reading or IDE.
So I update docs and comments.
Signed-off-by: Allen Sun<EMAIL_ADDRESS>

CI failing
0.08s$ GIT_CHECK_EXCLUDE="./vendor" TRAVIS_COMMIT_RANGE="${TRAVIS_COMMIT_RANGE/.../..}" make dco
git-validation -q -run DCO,short-subject,dangling-whitespace
 0fdf030 - FAIL - has whitespace errors. See `git show --check 0fdf030447098f12e6523d9c91e1a8fbcab3d7d3`.
1 commits to fix
make: *** [dco] Error 1

Oh, Thanks a lot for your remind. I noticed that in the original file there are some more whitespaces there. PTAL @AkihiroSuda

Codecov Report

Merging #1552 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1552   +/-   ##
=======================================
  Coverage   42.36%   42.36%
=======================================
  Files          24       24
  Lines        3368     3368
=======================================
  Hits         1427     1427
  Misses       1612     1612
  Partials      329      329

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20c6211...3c3cd21. Read the comment docs.


Custom workflow render error

is the result of rendering an iPXE script from a workflow. @dscherer posted this segment of the workflow tasks that matched that render:
"tasks": {
    "e58af2fe-ab88-497f-b321-d9777e999c77": {
      "definition": {
        "friendlyName": "Bootstrap Ubuntu",
        "injectableName": "Task.Linux.Bootstrap.Ubuntu",
        "implementsTask": "Task.Base.Linux.Bootstrap",
        "options": {
          "kernelFile": "vmlinuz-3.16.0-25-generic",
          "initrdFile": "initrd.img-3.16.0-25-generic",
          "kernelUri": "{{ api.server }}/common/{{ options.kernelFile }}",
          "initrdUri": "{{ api.server }}/common/{{ options.initrdFile }}",
          "basefs": "common/base.trusty.3.16.0-25-generic.squashfs.img",
          "overlayfs": "common/discovery.overlay.cpio.gz",
          "profile": "linux.ipxe",
          "comport": "ttyS0"
        },
        "properties": {
          "os": {
            "linux": {
              "distribution": "ubuntu",
              "release": "trusty",
              "kernel": "3.16.0-25-generic",
              "type": "microkernel"
            }
          }
        },

When I originally read the screenshot in iPXE, I thought it was a error rendering the iPXE script, but the results in on-http are clear, and show 200 responses.
[debug] [2016-03-08T16:27:28.287Z] [on-http] [Http.Server] [Server] http: POST 201 173.139 - /api/1.1/nodes/56dee34a28169ce56dfaaa19/workflows?name=Graph.Rubicon.Cataloger
 -> /lib/services/http-service.js:297
ipAddress: <IP_ADDRESS>
[debug] [2016-03-08T16:28:11.776Z] [on-http] [Http.Server] [faaa19] http: GET 200 33.028 - /api/common/profiles
 -> /lib/services/http-service.js:297
ipAddress: <IP_ADDRESS>
[debug] [2016-03-08T16:28:13.189Z] [on-http] [Http.Server] [faaa19] http: GET 200 77.596 - /api/common/profiles?macs=00%3A8c%3Afa%3Af1%3Acc%3Ac4
 -> /lib/services/http-service.js:297
ipAddress: <IP_ADDRESS>

My conclusion is that the iPXE script rendered out correctly, but that was an invalid iPXE script when it was done.
@benbp by chance, is that mechanism where this example is referencing options locally in the same task with the EJS/handlebars rendering template mechnism, i.e.
          "kernelFile": "vmlinuz-3.16.0-25-generic",
          "kernelUri": "{{ api.server }}/common/{{ options.kernelFile }}",

causing the issue? Is that value by chance not rendering at all? Or rendering as a blank line?

@dscherer broke into the iPXE script to get more detail:
monorail@monorail-micro:~$ curl -i  http://<IP_ADDRESS>:9080/api/common/profiles\?macs\=00%3A8c%3Afa%3Af1%3Acc%3Ac4
HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Content-Length: 2105
ETag: W/"839-Q18XFyxO6bAmy9CNT9uMNw"
Date: Tue, 08 Mar 2016 16:49:39 GMT
Connection: keep-alive

#!ipxe
set user-class MonoRail

echo
echo MonoRail Boilerplate iPXE...
set syslog <IP_ADDRESS>

# Interface that requested an IP originally.
set interface 00:8c:fa:f1:cc:c4

# Reboot Interval
set rebootInterval:int8 5

# Interface Search Index.
set ifIndex:int8 0
set ifIndexMax:int8 10

# Interface Boot Retries
set ifBootAttempt:int8 0
set ifBootAttemptMax:int8 5

# Close all the interfaces to begin with.
ifclose

# Iterate the interfaces to find a match by MAC Address.
:ifFind
iseq ${ifIndex} ${ifIndexMax} && goto ifFindFailure || goto ifFindContinue
:ifFindContinue
iseq ${interface} ${net${ifIndex}/mac} && goto ifBoot || inc ifIndex && goto ifFind

# Boot the found interface.
:ifBoot
iseq ${ifBootAttempt} ${ifBootAttemptMax} && goto ifBootFailure || goto ifBootContinue
:ifBootContinue
ifopen net${ifIndex}
ifconf net${ifIndex} && goto ifConfigured || inc ifBootAttempt && goto ifBoot

# Find Interface Failure
:ifFindFailure
echo Unable to locate interface ${interface}, restarting in ${rebootInterval} seconds.
sleep ${rebootInterval}
reboot

# Boot Interface Failure
:ifBootFailure
echo Unable to boot interface ${interface}, restarting in ${rebootInterval} seconds.
sleep ${rebootInterval}
reboot

# Configure the booted interface
:ifConfigured
route
echo MonoRail Boilerplate iPXE Completed.
echo
echo
echo Renasar: Error occurred while preparing your iPXE profile:
echo

        echo ejs:55

        echo     53| echo MonoRail Boilerplate iPXE Completed.

        echo     54| echo

        echo  >> 55| kernel <%=kernelUri%>

        echo     56| initrd <%=initrdUri%>

        echo     57| imgargs <%=kernelFile%> initrd=<%=initrdFile%> auto=true SYSLOGSERVER=<%=server%> API_CB=<%=server%>:<%=port%> BASEFS=<%=basefs%> OVERLAYFS=<%=overlayfs%> BOOTIF=01-<%=macaddress%> console=tty0 console=<%=comport%>,115200n8 <%=kargs%>

        echo     58| boot || prompt --key 0x197e --timeout 2000 Press F12 to investigate || exit shell

        echo

        echo kernelUri is not defined

echo
echo Booting to disk in ${rebootInterval} seconds...
sleep ${rebootInterval}
exit

It could be that that task had completed and it was a different task responsible for rendering at the time of the properties request?

@dscherer any detail what task was the pending one at the time of the render failure?
Also note - this is related to #154 in that the workflow perhaps should have failed when a template was attempting to render and failed


test(pageserver): add test interface to create artificial layers

Problem
This pull request adds necessary interfaces to deterministically create scenarios we want to test. Simplify some test cases to use this interface to make it stable + reproducible.
Compaction test will be able to use this interface. Also the upcoming delete tombstone tests will use this interface to make test reproducible.
Summary of changes

force_create_image_layer
force_create_delta_layer
force_advance_lsn

Checklist before requesting a review

[x] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

I don't feel timeline writer is the right thing to wrap for this test API. Timeline writer represents a normal+legal write to the timeline. The layers written by the timeline writer API goes through L0->compaction->L1. However, in my case, I want to construct a timeline state that is exactly what I want to test, instead of using high-level APIs to do a series of operations that may or may not end up in the state I want to test. As you see, the test cases are much simpler than before, and I don't see it would be a tech debt if this is only used in test cases.
Anyways, I feel what we lack is to describe a timeline state and directly test it in the test cases. For example, if I want to write a test "ensure vectored get does not go pass image layer", I will need to create two layers: an image layer above a delta layer. Using the current API, it's hard to achieve. It's hard to guarantee that after a series of compaction and data writing, the end state of the timeline is what I want. The interface to directly place a layer into the layer map is simple and reliable to recreate this scenario.


Curator errand fails to delete/purge old indices

We've configured our logsearch curator job to use the default retainment time of 7 days https://github.com/cloudfoundry-community/logsearch-boshrelease/blob/develop/deployment/logsearch-deployment.yml#L154-L161 . Unfortunately, the curator job doesn't appear to successfully run, leaving the elasticsearch data nodes to fill up to the VM's capacity & the nodes ends up filling to capacity.
To be specific, the job returns as having run successfully, but the logs show that the job didn't actually run successfully
Using deployment 'logsearch'

Task 1750

20:19:26 | Preparing deployment: Preparing deployment (00:00:01)
20:19:27 | Running errand: cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0)
20:19:27 | Running errand: maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0)
20:19:28 | Running errand: cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0) (00:00:01)
20:19:28 | Running errand: maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0) (00:00:01)
20:19:28 | Fetching logs for cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0): Finding and packing log files
20:19:28 | Fetching logs for maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0): Finding and packing log files
20:19:29 | Fetching logs for cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0): Finding and packing log files (00:00:01)
20:19:29 | Fetching logs for maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0): Finding and packing log files (00:00:01)

Started  Wed May  2 20:19:26 UTC 2018
Finished Wed May  2 20:19:29 UTC 2018
Duration 00:00:03

Task 1750 done

Exit Code  Stdout                                                                                Stderr
0          2018-05-02 20:19:28,079 INFO      Preparing Action ID: 1, "delete_indices"                                                                                -
           2018-05-02 20:19:28,084 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 7 days (based on index name), for logs- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
           2018-05-02 20:19:28,094 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
           2018-05-02 20:19:28,094 INFO      Action ID: 1, "delete_indices" completed.
           2018-05-02 20:19:28,094 INFO      Job completed.

~          2018-05-02 20:19:28,082 INFO      Preparing Action ID: 1, "delete_indices"                                                                                -
           2018-05-02 20:19:28,089 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 30 days (based on index name), for logs- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
           2018-05-02 20:19:28,523 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
           2018-05-02 20:19:28,523 INFO      Action ID: 1, "delete_indices" completed.
           2018-05-02 20:19:28,523 INFO      Job completed.

2 errands

Hi @aegershman! Curator job is supposed to run as regular long running job and triggered to cleanup old indexes using corn. It’s probably some missconfiguration in Curator job, that’s why it is available in bosh errands list. I will include the fix for this issue in the next release.
Sent from my iPhone
On May 2, 2018, 23:13 +0200, Aaron Gershman<EMAIL_ADDRESS>wrote:

We've configured our logsearch curator job to use the default retainment time of 7 days https://github.com/cloudfoundry-community/logsearch-boshrelease/blob/develop/deployment/logsearch-deployment.yml#L154-L161 . Unfortunately, the curator job doesn't appear to successfully run, leaving the elasticsearch data nodes to fill up to the VM's capacity & the nodes ends up filling to capacity.
To be specific, the job returns as having run successfully, but the logs show that the job didn't actually run successfully
Using deployment 'logsearch'
Task 1750
20:19:26 | Preparing deployment: Preparing deployment (00:00:01)
20:19:27 | Running errand: cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0)
20:19:27 | Running errand: maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0)
20:19:28 | Running errand: cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0) (00:00:01)
20:19:28 | Running errand: maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0) (00:00:01)
20:19:28 | Fetching logs for cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0): Finding and packing log files
20:19:28 | Fetching logs for maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0): Finding and packing log files
20:19:29 | Fetching logs for cluster_monitor/f93af0be-d234-4d05-afd5-eaaa3a35af57 (0): Finding and packing log files (00:00:01)
20:19:29 | Fetching logs for maintenance/87a40b9d-a6af-4943-b0b8-134d1bff02ff (0): Finding and packing log files (00:00:01)
Started  Wed May  2 20:19:26 UTC 2018
Finished Wed May  2 20:19:29 UTC 2018
Duration 00:00:03
Task 1750 done
Exit Code  Stdout                                                                                Stderr
0          2018-05-02 20:19:28,079 INFO      Preparing Action ID: 1, "delete_indices"                                                                                -
2018-05-02 20:19:28,084 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 7 days (based on index name), for logs- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2018-05-02 20:19:28,094 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-05-02 20:19:28,094 INFO      Action ID: 1, "delete_indices" completed.
2018-05-02 20:19:28,094 INFO      Job completed.
~          2018-05-02 20:19:28,082 INFO      Preparing Action ID: 1, "delete_indices"                                                                                -
2018-05-02 20:19:28,089 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 30 days (based on index name), for logs- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2018-05-02 20:19:28,523 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-05-02 20:19:28,523 INFO      Action ID: 1, "delete_indices" completed.
2018-05-02 20:19:28,523 INFO      Job completed.
2 errands
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thank you for the reply! When do you think the next release will be available? Do you have a recommendation on how we should handle the cleanup job until the next release?

@aegershman I want to update logsearch to use v6 of Elastic stack and then publish new release. Probably will try to finish by the end of the next week. But these dates may vary depending on my other activities.


Handle customer.subscription.updated webhook event type

Feature request

Extension name: firestore-stripe-subscriptions

Is your feature request related to a problem? Please describe.
When a user changes their subscription in the customer portal, currently the users subscription document remains with details of the original subscription. (I display current user basic plan info in my site without having to generate a customer portal link and then redirect etc driven from the firestore document)
Describe the solution you'd like
When a user changes their subscription in the customer portal, the subscription document should be updated to reflect the new subscription details.
Describe alternatives you've considered
Custom function myself to handle this webhook event, happy to do this an merge upstream. Not sure if there were any plans for this event type.
Stripe api directly to receive the current subscription information.

It does: https://github.com/stripe/stripe-firebase-extensions/blob/next/firestore-stripe-subscriptions/functions/src/index.ts#L579
Do you get any logs in your Stripe Dashboard for the customer.subscription.updated events? Also check the logs of the Firebase functions, they might give you a hint what's wrong in your configuration.

Ah yes, I found an error with the price app ID which was causing the update to fail. ! :+1:

It does: https://github.com/stripe/stripe-firebase-extensions/blob/next/firestore-stripe-subscriptions/functions/src/index.ts#L579
Do you get any logs in your Stripe Dashboard for the customer.subscription.updated events? Also check the logs of the Firebase functions, they might give you a hint what's wrong in your configuration.

This link no longer works.  I'm trying to figure out how to determine if the customer paused their subscription and I see the subscription update webhook is processed in the logs however I see no change in firestore that shows it paused.  Any suggestions?


Perform some tests with  Python Multithreading without GIL

As of 2021, CPython implementation still uses the infamous "global interpreter lock (GIL)".
This is the design point behind PyAF using the multiprocessing module and not multithreading.
Now, there is a POC to remove the GIL for CPython (thanks to @colesbury ):
https://github.com/colesbury/nogil
The goal of this issue is to perform some initial tests with nogil and PyAF.
Not sure if this will help with the upcoming PyAF release (Probably later).

Pre-built Docker image :
https://hub.docker.com/r/colesbury/python-nogil


Platypus 5.4.1 bug: it does not respect/read Text Window colours set

Hi @sveinbjornt thank you for keeping Platypus alive!
I have tried twice to generate an application with latest Platypus 5.4.1 and it seems that despite setting the text window background and foreground colours, they are not used/respected even if they are declared in AppSettings.plist generated.
For example, setting BLACK background window and GREEN foreground for text, I still get white background and purple text in the resulting generated app... Here is what is in the generated PLIST:
	<key>TextBackground</key>
	<string>#000000</string>
	<key>TextFont</key>
	<string>Monaco</string>
	<key>TextForeground</key>
	<string>#00ff00</string>
	<key>TextSize</key>
	<real>13</real>

Moreover, I don't see any longer the encoding, was it removed? Perhaps not set via the font selector window, not sure:
	<key>TextEncoding</key>
	<integer>4</integer>

Thank you for having a look at this bug!

Apologies @sveinbjornt I corrected my title, it has to do with 5.4.1 version.
For some reason, you mention 5.4.2 in your documentation but never released it...
https://github.com/sveinbjornt/Platypus/blob/master/Documentation/Documentation.md

I can replicate the same bug in Platypus 5.4.0 as well.
However, in that build, the font size is not real but integer?
	<key>TextSize</key>
	<integer>14</integer>

Is this confirmed to be correct for either 5.4.0 and 5.4.1?

Dark mode inverts the colours.

Thanks @sveinbjornt I will try to invert the inverted colour then 😄


Broken links

Details in commit messages. Resolves #7 and #8, because #8 finally got on my nerves enough when I wanted to go look at some numbers.

Always good to look at warnings. That particular one for /switchboard/static/fix-links.js is expected in dev. In production, the expected warning will be for /static/fix-links.js. Rationale is in 4342a46 (which is for our CSS, but it applies the same to the JS).

Always good to look at warnings. That particular one for /switchboard/static/fix-links.js is expected in dev. In production, the expected warning will be for /static/fix-links.js. Rationale is in 4342a46 (which is for our CSS, but it applies the same to the JS).

That makes sense. Thanks!

Hit a bug after deployment which I fixed in https://github.com/seattleflu/scan-switchboard/commit/ed7b02623d19f6b41ad7fb766b2bdb736a7b8cb6.


[qpid-proton] Stop using linker optimization

These optimizations require the gold linker on Linux, and this linker is not typically available.
I've also reordered alphabetically CMake configuration's options for the qpid-proton port.
Describe the pull request

What does your PR fix?
Fixes #24128

Which triplets are supported/not supported?

Yes

Cool, thanks!


Batch Daemon not Running Leads to time-outs when Message Queue is full

We are testing with the Batch Daemon to do logging, tracing and possibly some PubSub requests (and more later). Currently, we are looking into the different failure modes that could happen and how we should respond to those.
One of the failure modes is that the Batch Daemon might not be running. At this moment, Batch Daemon not running would mean would mean that requests would be queued through the message queue. This queue has two problems though, on a potential server restart we would lose those messages, and next to that the queue can get full, at which moment the request will be waiting for the message queue to have space again and will eventually time-out.
We were thinking about increasing the message queue size to increase the time we could handle without Batch Daemon running, but this would increase the number of messages that we could lose in case of a server failure. And in case of a longer failure of Batch Daemon, the queue would still fill-up.
Another possible solution would be to make it configurable what you would want to happen if the message queue is full. For example, adding these two options:

Throwing an exception, so you can catch that and find another mean to do the request (it seems that making msg_send() has a $blocking variable that is by default true. If that would be configurable to false it would start writing to a file and send that to the queue, and in the end throw an exception if that also failed.
Doing the request directly

Any other solutions to the problem would of course also be welcome.
Relates to https://github.com/googleapis/google-cloud-php/issues/683

Closing as seems fixed by #3176


ENH: add stirling2 function to scipy.special

Reference issue
Closes gh-17890
What does this implement/fix?
Implements a ufunc in scipy.special for Stirling numbers of the second kind, in scipy/special/cephes.
Additional information
The method follows the discussion in the linked issue.

it looks like the windows builds are failing:
2023-03-06T02:27:11.4935123Z error: Command "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -D_USE_MATH_DEFINES -ID:\a\1\s\scipy\special -IC:\hostedtoolcache\windows\Python\3.9.13\x86\lib\site-packages\numpy\core\include -IC:\hostedtoolcache\windows\Python\3.9.13\x86\include -Iscipy\_lib -Iscipy\_lib\boost_math\include -Iscipy\_build_utils\src -IC:\hostedtoolcache\windows\Python\3.9.13\x86\lib\site-packages\numpy\core\include -Ibuild\src.win32-3.9\numpy\distutils\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\cppwinrt -IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE -IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\winrt /Tcscipy\special\cephes\stirling.c /Fobuild\temp.win32-3.9\scipy\special\cephes\stirling.obj /arch:SSE2" failed with exit status 2
2023-03-06T02:27:11.4939270Z scipy\special\cephes\stirling.c(32): error C2057: expected constant expression
2023-03-06T02:27:11.4940435Z scipy\special\cephes\stirling.c(32): error C2466: cannot allocate an array of constant size 0
2023-03-06T02:27:11.4941418Z scipy\special\cephes\stirling.c(32): error C2133: 'prev': unknown size
2023-03-06T02:27:11.4942535Z scipy\special\cephes\stirling.c(33): error C2057: expected constant expression
2023-03-06T02:27:11.4943476Z scipy\special\cephes\stirling.c(33): error C2466: cannot allocate an array of constant size 0
2023-03-06T02:27:11.4944425Z scipy\special\cephes\stirling.c(33): error C2133: 'curr': unknown size
2023-03-06T02:27:11.4945275Z INFO:

I suspect this is because I'm using const int arraysize = n+1; as the array size inside the function.  This works fine for me locally when I include the c99 flag to gcc but I'm using a mac and I don't have access to a windows machine. Is there something different that needs to be done for the windows compilers to handle variable length arrays in windows?

const int arraySize = n + 1;

I suspect this is because I'm using const int arraysize = n+1; as the array size inside the function. This works fine for me locally when I include the c99 flag to gcc but I'm using a mac and I don't have access to a windows machine. Is there something different that needs to be done for the windows compilers to handle variable length arrays in windows?

The const here is wrong - the value must be constant for the duration of the program, and here it depends on n which is a parameter to the function. So just drop the const I'd say. Why it's different across platforms is probably just because it's undefined behavior.
Also note this failure:
 Exception: /home/runner/work/scipy/scipy/scipy/special/tests/test_stirling.py is not installed

You can fix it by adding the new file to the listing in scipy/special/tests/meson.build.

The const here is wrong ... So just drop the const I'd say. Why it's different across platforms is probably just because it's undefined behavior.

Got it, removed const in latest commit e6bb72d...

You can fix it by adding the new file to the listing in scipy/special/tests/meson.build.

Done in commit e6bb72d...

I switched to heap memory from @steppi 's comments (and free) before returning. I ran the new version through valgrind locally:
:==12039==
==12039== HEAP SUMMARY:
==12039==     in use at exit: 0 bytes in 0 blocks
==12039==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==12039==
==12039== All heap blocks were freed -- no leaks are possible
==12039==
==12039== Use --track-origins=yes to see where uninitialised values come from
==12039== For lists of detected and suppressed errors, rerun with: -s

so I think everything should be good from memory leak perspective AFAICS.
All returned values look the same as using VLAs (this is good).
Still not sure what we want to do about the overflow issue (large n or k input) though?

FWIW it looks like the failing windows builds are the same issue that Tyler opened https://github.com/scipy/scipy/issues/18108

Actually, I'm pretty frightened by having all these allocations in the inner loop of a ufunc. This stackexchange answer seems to give an efficient solution based on the inclusion exclusion based formula. As for overflow, could we have the function take ints as input but return a long so we can handle larger values?

Actually, I'm pretty frightened by having all these allocations in the inner loop of a ufunc. This stackexchange answer seems to give an efficient solution based on the inclusion exclusion based formula.

I had initially looking into IE-check  the issue, in practice IE doesn't seem to be isn't faster because the runtime ignores the cost of computing the binomial coefficients-one for each entry from i=1 to k. As for the memory allocations, it's really only 1 per call which doesn't seem too bad to me.
The alternative is to do everything in pure python and compute up through an array of (n,k) values. See this comment from the original issue for that computing values approach. Another benefit of the pure python approach is greater precision (overflow is less of a concern) if we stay with the python int type.

The algorithm from the above link uses a recurrence to avoid recomputing the binomial coefficient at each step and uses fewer overall operations than the allocation heavy algorithm.

Your IR based implementation from this gist is recomputing the binomial coefficient from scratch at each iteration instead of using the recurrence from the above link to update it with one multiplication and one division at each step. The algorithm from the link uses fewer overall options than the allocation heavy approach.

I've come with a plan that might work.

Ok. FWIW the calculation will overflow long before this point so mayvbe it makes sense to raise an error if n > X where X is the value of overflow (17 on my machine w/int being 32 bit IIRc)?

OK, so the size of an int is platform dependent, but for compatibility uses it usually will be 32 bits, even on 64 bit machines.
$${2000000 \brace 1999999} = {2000000 \choose 2} =<PHONE_NUMBER>000$$
won't overflow when using 64 bit ints, but would overflow when using 32 bit ints. We will want to support 64 bit. The current special infrastructure allows types int and long. On 64 bit linux a long is 64 bits, but on windows it's still 32 bits. We could either support both int and long, maybe using Cython with fused types, or just support long.
I still think we should be using the fast IE based algorithm from here, not the allocation based one. If one wants to compute many Stirling numbers, the allocation based one will duplicate too much effort, so isn't very efficient actually.
For overflow. I think we can check for it inside the compiled function and return -1 if there's been overflow. We can also have a pure python implementation which can handle arbitrary sized ints. There would be a Python wrapper, which first calls the vectorized ufunc. We then check for -1s. If any exist we convert the array to type object, and use the pure Python implementation to fill the -1s with the correct values.
Thoughts?

Oh, I see. Even when speeding up the binomial coefficient calculation, the IE formula is still unworkable because you need to divide by factorial at the end, and the numerator and denominator will both overflow even for reasonable input.

the IE formula is still unworkable ... overflow

Yes. I noticed this too, the -1s in the IE formulate indicate that something needs to be too large (at some point) so that overflows are likely to occur sooner. If arbitrary precision (gmp or pure python) you don't have this concern.

@steppi Latest commit has your memory optimization-using only 1 array. I'll think a bit more on the overflow checking.
It seems like a check for curr[j] < 0 in the inner for loop would work but I haven't tested it out yet so I'll maybe do those over the weekend.

@rlucas7, I thought it would be good to also support arbitrarily large ints using a pure Python implementation. I've submitted a nested PR https://github.com/rlucas7/scipy/pull/1 to make stirling2 work similarly to scipy.special.comb. It is unfinished, it still needs an implementation of an asymptotic approximation for the exact=False case, is in need of updated documentation, and probably needs some polishing. Please take a look and let me know what you think.

I think it's important to keep the API consistent from the start and not merge this without an exact keyword arg. Do you think it would be better to have the limited exact=False behavior from rlucas7#1 or to raise a NotImplementedError like for factorialk and then implement the exact=False case in my follow up PR with the asymptotic approximations? I'll consider your judgement final here.

Actually, @rlucas7, if your preference is to simply raise NotImplementedError when exact=False and only include the pure Python implementation in this PR, I think that's acceptable.

I think it's important to keep the API consistent from the start and not merge this without an exact keyword arg. Do you think it would be better to have the limited exact=False behavior from rlucas7#1 or to raise a NotImplementedError like for factorialk and then implement the exact=False case in my follow up PR with the asymptotic approximations? I'll consider your judgement final here.

Actually, @rlucas7, if your preference is to simply raise NotImplementedError when exact=False and only include the pure Python implementation in this PR, I think that's acceptable.

Thanks for the ping. I haven't reviewed this PR in detail, but from a glance, it seems it restricts itself to int64 outputs?
The exact keyword for the factorials etc. is owed to the fact that we need to make some trade-offs w.r.t. speed & accuracy; ideally, we wouldn't need that keyword, but different users need different things; some need performance, some absolutely want integers (hence using a python implementation with unlimited integers, rather than falling using the faster implementation that yields floats).
This PR is in a somewhat different situation, because it only implements the intersection of those cases (i.e. always integer, always fast). From my POV, whether the an exact-keyword is added now (with a NotImplementedError) or later (when a pure python implementation is added) is therefore secondary. I'd have a preference for the former, but if and only if we are convinced/committed that this pure python implementation will materialize. Otherwise, we're just promising something that might stay unimplemented for years, which is not a good look.
One other thing that stands out from an API consistency POV, is that we have a much earlier cut-off based on whatever fits into int64; in the factorial family, that doesn't happen until the much larger float64 max, and not at all for the exact=True case. It might be worthwhile (as a future improvement perhaps) to not throw overflow errors so quickly, but fall back to some approximation.
Finally, I'd like to see test coverage of the case of too large inputs. For reference, the factorial functions (with exact=False) return inf, not an error (I'm not saying that's necessarily the best choice, but it has been like that for a long time).

Thanks for the summary. Sorry that I missed the dependence on another PR to this PR.

Should pure Python implementation not have an exact keyword arg at all, or should it have one that raises a notImplementedError? Once we get this current PR in shape, I'll ping you again for a review.

Given that you've planned the exact=False approximation already (which makes me confident we can actually do it, rather than just claim it's possible), I'd say add the exact= keyword and raise NotImplementedError until then.
BTW, I also agree that switching over to python integers on overflow is not a good approach (fragile, surprising behaviour changes based on implementation details, etc.). I'd prefer to keep the duck-typed-integer & fast-but-potentially-float worlds separate. This would also match what the factorial functions are doing.

Thanks for the summary. Sorry that I missed the dependence on another PR to this PR.

No worries. Thanks for the input.

Given that you've planned the exact=False approximation already (which makes me confident we can actually do it, rather than just claim it's possible), I'd say add the exact= keyword and raise NotImplementedError until then.
BTW, I also agree that switching over to python integers on overflow is not a good approach (fragile, surprising behaviour changes based on implementation details, etc.). I'd prefer to keep the duck-typed-integer & fast-but-potentially-float worlds separate. This would also match what the factorial functions are doing.

Yeah, agreed on both points. Thanks!

The failing test is because I calculated with mpmath which defaults to a non-exact calculation routine. Here is example:
>>> import mpmath
>>> mpmath.stirling2(25,17)
mpf('48063331393110.0')
>>> mpmath.stirling2(30,4)
mpf('48004081105038304.0')
>>> mpmath.stirling2(30,4,exact=True)
48004081105038305
>>>

The last test also failed on the 25,17 instance and I must've mistyped the last digit somehow.
The value now matches mpmath and what actually comes out of stirling2 here:
>>> import mpmath
>>> mpmath.stirling2(25,17)
mpf('48063331393110.0')
>>> mpmath.stirling2(25,17, exact=True)
48063331393110
>>> mpmath.stirling2(25,17, exact=False)
mpf('48063331393110.0')
>>> import scipy.special as sc
>>>  sc.stirling2(25,17)
array(48063331393110)

@rlucas7 Wow, it looks like did the thing where it works over arrays with a dynamic programming type algorithm while avoiding duplicate work. That's pretty cool. I've just been through a busy few months but have more time now. I plan to look at this closely early next week.

@rlucas7 Also, can you merge main into your branch? There have been some CI changes.

~@rlucas7 Also, can you merge main into your branch? There have been some CI changes.~ Nevermind, I forgot that main automatically gets merged before CI is run. I thought not having the latest commits might have been behind linting failing to work. Not sure what's going on.

I'll do an rebase -i master before doing more changes, maybe that will fix. Probably wont get to this until upcoming Sunday at earliest though.

~@rlucas7 Also, can you merge main into your branch? There have been some CI changes.~ Nevermind, I forgot that main automatically gets merged before CI is run. I thought not having the latest commits might have been behind linting failing to work. Not sure what's going on.

I'll do an rebase -i master before doing more changes, maybe that will fix. Probably wont get to this until upcoming Sunday at earliest though.

No need.  I didn’t see them before posting my review. Sorry for stepping on your toes. It looks like our comments largely agree though.

@rlucas7 it looks like you posted your replies to @jorenham slightly before me. I didn’t see them before posting my review. Sorry for stepping on your toes. It looks like our comments largely agree though.

Yep, no worries. I'll update binom -> comb in the docstring on my local changes. One thing I'm not clear on is whether we want the behavior to return an int or an array with a single object in the scenario:
>>> stirling2(5, 1)
1
>>> stirling2([5], [1])
array([1], dtype=object)

I wasn't sure from an earlier comment if we wanted the behavior for the call stirling2([5], [1]) to be an int or a 1d array, can you clarify?
Also, using a couple larger values to confirm handling w/the overflow of 64bit int case and confirmed w/mpmath,
>>> import mpmath
>>> mpmath.stirling2(42,17,exact=True)
2801934359500572414253157841233849412
>>> mpmath.stirling2(42,17,exact=True) ==<PHONE_NUMBER>500572414253157841233849412
True
>>> mpmath.stirling2(43,23,exact=True) ==<PHONE_NUMBER>2277144547280648984426251
True
>>> mpmath.stirling2(42,17,exact=True) == stirling2(42, 17)
True
>>> mpmath.stirling2(43,23,exact=True) == stirling2(43, 23)
True

The latest commit cleans up flake8 issues on the lines added in this commit-there's a bunch of pre-existing flake8 errors in _basic.py and tests/test_basic.py that are on other parts of these 2 modules that I'm leaving as is.
I also added a couple tweaks to the unit test cases and docstrings to exhibit (docstring) and test (unit tests) the behavior when we have floats. The docstring already indicated-in writing-that we truncate floats, and now we have tests for that behavior too.
I swapped out binom with comb in the see also section as discussed above in review.
Keeping the behavior on negative inputs to return 0 as also discussed in review.
Also added a couple examples to tests where int64 would overflow. I checked them against mpmath to confirm the values match-they do.
There is still one issue I noticed-I'll put it in a separate comment to not mix things.

I wasn't sure from an earlier comment if we wanted the behavior for the call stirling2([5], [1]) to be an int or a 1d array, can you clarify?

stirling2([5], [1]) should output a 1d array.  stirling2(5, 1) should be an int. stirling2(np.array(5), np.array(1))can also just be anint` instead of a 0d array like the input arrays. This seems to be the expected 0d array behavior for ufuncs in special; it also is much easier to implement.

One thing we don't have a test for is cases with negative inputs between -1 and 0. In these cases the behavior of int(-0.3) (say) is to round to 0. So it technically is not matching the part of the docstring that says negative values are 0 because the inputs are truncated to a valid input range. We could discard the inputs if either n or k entries are less than 0 to get the behavior of 0s for these but the documentation is somewhat ambivalent in this scenario (between truncating floats and negative inputs).
@steppi What do we think is the desired behavior here?
In either case I think we should have a single sentence which specifies the behavior in this scenario.
I'll also add an (n,k) pair in the doctest to exhibit the desired behavior (once we agree on what that is exactly).
My inclination is to ignore/drop all (n, k)-tuples where either n or k is negative. Effectively making the behavior of (-0.3, 0) return 0 rather than 1 which it would currently indicate.
E.g.
>>> stirling2(-0.3,0)
1
#and
>>> stirling2(-0.3,-0.3)
1

instead do
>>> stirling2(-0.3,0)
0
#and
>>> stirling2(-0.3,-0.3)
0

and current behavior for things like
>>> stirling2(-1.3,-0.3)
0

stays the same.

tirling2(np.array(5), np.array(1))

I wasn't sure from an earlier comment if we wanted the behavior for the call stirling2([5], [1]) to be an int or a 1d array, can you clarify?

stirling2([5], [1]) should output a 1d array. stirling2(5, 1) should be an int. stirling2(np.array(5), np.array(1)) can also just be an int instead of a 0d array like the input arrays. This seems to be the expected 0d array behavior for ufuncs in special; it also is much easier to implement.

Got it. This matches the current behavior (as of commit c5530f5)
so we should be fine as it.

One thing we don't have a test for is cases with negative inputs between -1 and 0. In these cases the behavior of int(-0.3) (say) is to round to 0. So it technically is not matching the part of the docstring that says negative values are 0 because the inputs are truncated to a valid input range. We could discard the inputs if either n or k entries are less than 0 to get the behavior of 0s for these but the documentation is somewhat ambivalent in this scenario (between truncating floats and negative inputs).
@steppi What do we think is the desired behavior here?
In either case I think we should have a single sentence which specifies the behavior in this scenario. I'll also add an (n,k) pair in the doctest to exhibit the desired behavior (once we agree on what that is exactly).
My inclination is to ignore/drop all (n, k)-tuples where either n or k is negative. Effectively making the behavior of (-0.3, 0) return 0 rather than 1 which it would currently indicate.
E.g.
>>> stirling2(-0.3,0)
1
#and
>>> stirling2(-0.3,-0.3)
1

instead do
>>> stirling2(-0.3,0)
0
#and
>>> stirling2(-0.3,-0.3)
0

and current behavior for things like
>>> stirling2(-1.3,-0.3)
0

stays the same.

OK, now we're running into trouble. I'll have to think about this. Rounding in comb was an annoyance which has been deprecated, see https://github.com/scipy/scipy/pull/15359, https://github.com/scipy/scipy/issues/15722, and https://github.com/scipy/scipy/issues/15722. I don't think we should be rounding here.
If stirling2 were just a scalar function, we could raise an error for non integral arguments, but I think the ufunc behavior you've implemented is very useful. This is a situation where returning nan could make sense. @h-vetinari, do you have any thoughts here?

If stirling2 were just a scalar function, we could raise an error for non integral arguments, but I think the ufunc behavior you've implemented is very useful. This is a situation where returning nan could make sense. @h-vetinari, do you have any thoughts here?

I think we can put np.Nan into the array with no issues actually, it works b/c object is very general :) . IIUC this does somewhat differ from how comb works, so let's see what @h-vetinari thinks.

Actually, I think it might be fine to raise an exception for float input here. That's what happens when giving complex input to a ufunc which only accepts real input, or when giving string input when only numeric is expected.
import scipy.special as sc

sc.zeta([1 + 2j, 2, 3])

yields
TypeError: ufunc '_riemann_zeta' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''.

(It's an open issue that our zeta implementation doesn't support complex input gh-9182).
Also it turns out that the current implementation of stirling2 in this PR is veering off into JavaScript land in terms of type coercion.
stirling2("100", "2")
#<PHONE_NUMBER>14114700748351602687

We should be raising an exception for string input and I'd support raising an exception for float input as well since there's precedent for such things.

@steppi in the latest 2 commits (90b0d434 and aa6951a) I made it so the input arrays must be integers (of some precision) and raise TypeError on the other input types. The type checking can occur before any computation happens in the DP so we don't waste any time computing if the types aren't correct.
I also added some clarification in the docstring 'notes' section so that it is explicitly stated that inputs must be all integer types (and that any negative argument value will result in a 0 value returned.
Of course let me know if you've got a better idea of how to achieve the type check.

@h-vetinari, I’m confident enough in the math and don’t think this needs another in depth technical review. I’m more interested in your thoughts on things like the behavior and API consistency with other combinatorial functions in special. e.g. is it ok to raise an exception for non-integer inputs?
let me know if

you don’t have bandwidth at the moment but would like to have a chance to give this a stamp of approval before it goes in.

you don’t have bandwidth at the moment and would be ok if I ask someone else to give a final stamp of approval.

you’ll have time to get to it in the next couple weeks or so

Thanks @h-vetinari; good suggestions.

I had switched off notifications for this PR as I was getting too many emails sweat_smile

Ha, yeah, this one was pretty active at times.

In terms of consistency with the combinatorial (resp. factorial) functions, it's worth noting that they're1 using exact=False, i.e. the default is fast, and accuracy is an explicit user opt-in.
Here we're introducing the function with a default of exact=True, which -- strictly speaking -- would need a deprecation cycle if we ever want to change it in the future (unless the approximation gets implemented during the 1.12 cycle and we switch to exact=False before release).
I don't think this is actionable for this PR, i.e. we can merge this and if the approximation arrives after 1.12, we'll just have to do a deprecation cycle to switch the default to exact=False.

I think we'll need to wait until the accuracy of the approximation approaches machine precision to make it the default. I've implemented second order asymptotic expansion from Temme's Asymptotic Methods for Integrals but iirc I found the relative error is typically no better than 1e-10. To go a higher order expansion, I'll need to work out the terms using the recipe Temme provides, but then it might be difficult to find a qualified reviewer with sufficient bandwidth to check the math. Then again, we could also just test it near exhaustively.
For the future, do you think the second order expansion with middling accuracy is worth making available as a non-default, or should we wait until we have something near machine precision?

I've implemented second order asymptotic expansion from Temme's Asymptotic Methods for Integrals but iirc I found the relative error is typically no better than 1e-10.

I think an average relative accuracy of 1e-10 is plenty for most usecases, and where it's not, people can use exact=True.
The question is what the worst case relative accuracy (in a reasonable value range) is. If it's also near 1e-10 (and not drastically worst like 1e-4), then I'd much prefer to have the approximation sooner, so we can release with a default of exact=False.
If you can later improve the approximation by going to higher order (in a way that doesn't regress performance too much), then we could land that without having to deprecate anything.

Results are actually worse than I remembered. Here's the maximum of $log_{10}$ relative error for each value of $N$ for the second order expansion. The kink in the graph happens when the output starts to overflow for some values of $K$. I think we should go to higher order and am pretty sure I can get it done before we branch for 1.12.
(For smaller values of $N$ we can do an exact calculation.)

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy.special as sc

def stirling2_asymptotic(n, k):
    mu = k/n
    k, n = float(k), float(n)

    def f(x):
        return mu * x - 1 + np.exp(-x)

    def df(x):
        return mu - np.exp(-x)

    def d2f(x):
        return np.exp(-x)

    delta = 1/mu * np.exp(-1/mu)
    x0 = 1/mu + sc.lambertw(-delta)
    t0 = (n - k) / k
    F = np.sqrt(t0/((1 + t0)*(x0 - t0)))
    A = -n*np.log(x0) + k*np.log(np.exp(x0) - 1) - k*t0 + (n - k)*np.log(t0)
    try:
        result = np.exp(A)*k**(n - k)*F*sc.binom(n, k)
    except OverflowError:
        result = np.inf
    return result.real

def stirling2_asymptotic_order2(n, k):
    mu = k/n
    k, n = float(k), float(n)

    def f(x):
        return mu * x - 1 + np.exp(-x)

    def df(x):
        return mu - np.exp(-x)

    def d2f(x):
        return np.exp(-x)

    delta = 1/mu * np.exp(-1/mu)
    x0 = 1/mu + sc.lambertw(-delta)
    t0 = (n - k) / k
    F = np.sqrt(t0/((1 + t0)*(x0 - t0)))
    A = -n*np.log(x0) + k*np.log(np.exp(x0) - 1) - k*t0 + (n - k)*np.log(t0)
    F1 = (
        -2*x0**3 + 2*t0**5 + 4*t0**3 + 4*t0**4 + 3*x0**2 * t0 - 6*x0*t0**4
        - 5*x0**2 * t0**2 + 2*x0**4 * t0 + x0**3*t0 - 6*x0**3 * t0**2
        + 8*x0**2 * t0**3
    ) / (24*F*(1 + t0)**2*(x0 - t0)**4)
    try:
        result = np.exp(A)*k**(n - k)*sc.binom(n, k)*(F - F1/k)
    except OverflowError:
        result = np.inf
    return result.real

rows = []
for n in range(1, 350):
    for k in range(2, n):
        exact = sc.stirling2(n, k)
        asymptotic = stirling2_asymptotic(n, k)
        asymptotic2 = stirling2_asymptotic_order2(n, k)
        try:
            rel_error = abs(float(exact) - asymptotic) / abs(float(exact))
            order2_rel_error = abs(float(exact) - asymptotic2) / abs(float(exact))
        except OverflowError:
            continue
        rows.append(
            {
                "n": n,
                "k": k,
                "exact": exact,
                "approx1": asymptotic,
                "approx2": asymptotic2,
                "rel_error1": rel_error,
                "rel_error2": order2_rel_error,
            }
        )

df = pd.DataFrame(rows)

result = df.groupby("n", as_index=False)["rel_error2"].max()

fig, ax = plt.subplots()
ax.plot(result.n, np.log10(result["rel_error2"]))
ax.set_xlabel("N")
ax.set_ylabel("Max log10 relative error")

I think we should go to higher order and am pretty sure I can get it done before we branch for 1.12.

OK, I trust you on that. And if it doesn't land, it's not the end of the world either, just means we have to do the deprecation cycle.
I actually have an idea of how we could avoid deprecation even then, but we can discuss that closer to 1.12 branching off, when we'll know whether the approximation has landed or not. If we merge this PR, could you open an issue milestoned for 1.12 about adding the approximation? That way we won't forget it, and will have a place to discuss.

If we merge this PR, could you open an issue milestoned for 1.12 about adding the approximation? That way we won't forget it, and will have a place to discuss.

Will do.

The failures are unrelated. @h-vetinari, if you think this looks good now I'll merge today and make an issue for the exact=False case.

@h-vetinari, if you approve I'll merge now and PR what I have for exact=False within one week, and then try to work on adding more terms to the asymptotic expansion soon enough for someone to review before we branch for 1.12.

I tried restarting the job because I'm wondering about the more extensive (than usual) failures. I'd like to get this a bit greener before we merge. I also see there's a conflict with main that needs to be fixed.

I tried restarting the job because I'm wondering about the more extensive (than usual) failures. I'd like to get this a bit greener before we merge. I also see there's a conflict with main that needs to be fixed.

Merge conflict fixed; I'd missed gh-18624. The failures in test_hermitian_mode were appearing everywhere and are definitely unrelated.

The failures in test_hermitian_mode were appearing everywhere and are definitely unrelated.

I don't doubt it, my point was that other recent PRs have had basically 100% green CI, so I'd like to spend a bit more time on figuring out what's going on in this PR rather than (potentially) making it everyone's problem.

The failures in test_hermitian_mode were appearing everywhere and are definitely unrelated.

I don't doubt it, my point was that other recent PRs have had basically 100% green CI, so I'd like to spend a bit more time on figuring out what's going on in this PR rather than (potentially) making it everyone's problem.

Huh. Might be a transient issue. Here is is popping up less than an hour ago https://github.com/scipy/scipy/actions/runs/5350449116/jobs/9703062620?pr=18728.

Same failure in test_hermitian_mode. It's also failing for every recent PR  https://github.com/scipy/scipy/pulls. It looks like no one has made an issue related to the failure yet. I'll have time to look into it tomorrow if no one else gets to it by then.

Same failure in test_hermitian_mode. It's also failing for every recent PR https://github.com/scipy/scipy/pulls. It looks like no one has made an issue related to the failure yet. I'll have time to look into it tomorrow if no one else gets to it by then.

Thanks for continuing to shepherd this one @steppi let me know if there's more you'd liek me to do, e.g. I can rebase off master/main if we think that will maybe fix some things.
This latest push from me is just a few small changes to the docstring b/c I noticed the braces weren't getting rendered correctly.
For me these 4 lines in latex
\documentclass{article}
\begin{document}
${n \brace{k}}$
\end{document}

compile into a pdf as

I also aligned the TypeError message for K to match the changes @h-vetinari suggested for the N case.
I followed the syntax for binom references b/c those were also getting a bit mangled in this pr but the doc build for binom looks good https://output.circle-artifacts.com/output/job/8bac28a8-603b-426e-9aa6-8371b2ae3c6c/artifacts/0/html/reference/generated/scipy.special.binom.html#r2d6855f23738-1
let me know if there's anything else I can do on this one to help out in getting this over the line

Thanks @rlucas7. I think it's almost there.

I followed the syntax for binom references b/c those were also getting a bit mangled in this pr but the doc build for binom looks good

The binom docstring is a raw string, so that's why the build looks fine.
https://github.com/scipy/scipy/blob/876c68da9dfa6d91f110c7f85d621a9177b89eb0/scipy/special/_add_newdocs.py#L808-L814
If the latest doc build still fails I think we can just turn the stirling2 docstring into a raw string

Docs failures have been resolved. The arpack test, test_hermitian_mode, is still failing for Linux Meson tests / Meson build (3.11, false) (pull_request); I plan to look into that. The other failure in Linux Meson tests / Prerelease deps and 64-bit BLAS (3.9) (pull_request) is gh-18721.

The arpack test, test_hermitian_mode, is still failing for Linux Meson tests / Meson build (3.11, false) (pull_request); I plan to look into that.

Hopefully this can help to get started https://github.com/scipy/scipy/issues/18736  :)

@steppi I worked through Temme's paper this weekend and reproduced the results in that paper. In particular his equation 4.4 on page 10 matches what you have here in the details code dropdown. I reproduce the max error values with my own implementation (I missed the code in the dropdown tab in your comment) so I independently reproduced both the code and the max error values Temme has noted for n=10, 20, etc for the second order approximation. I think that's what we should use here-unless you've got a more accurate one. The choice of N where we use the approximation over the exact value needs to be made. It seems with the second order the max relative error for N=50 is already 0.00002 at m=7 and for N=100 at m=13 with max relative error of 4.7973885812281365e-06 so probably something at or above N=100 is fine. For larger values of N:
(max rel error, K that achieves max error, N)
(1.1865872292929588e-06, 26, 200)
(9.36707897293228e-07, 29, 225)
(1.62298310561457e-07, 20, 250)
(2.1612156823753847e-08, 14, 275)
(5.521646313770846e-11, 11, 300)
so somewhere between 275 and 300 seems ok  as a threshold.
incidentally if you want me to look at a pr for the Temme approximation just drop me an email once it's up. I'm comfortable reviewing a PR on that now.
Here is the code I used to generate the values noted here:
from scipy import optimize
from math import sqrt, comb
## calculated via exact method using dp soln above...
# setting n=10 we can verify that max relative error occurs @ m=4 w/value =0.000466 (rounds to value given in Temme paper)
# also n=20 verifies max relative error in paper too @ m=3 w/value = (0.000116, rounds to 0.00012 in paper)
n = 275
actual = stirling2(n, range(n+1))
m2 = m1 = (-np.inf, -1)
for m in range(1, n):
    # calculate t0 as described on page 235 of temme
    # this is noted in the writing just after equation 2.6 on page 235
    t0 = (n-m)/m
    def func(x):
        return (m/n)*x - 1 + np.exp(-x)
    # calculate x0 by solving root finding problem
    # equation 2.5 in Temme
    res = optimize.root(func, n/m, method='broyden1', tol=1e-14)
    x0 = res.x.take(0)
    # now
    def phi(x):
        return -n * np.log(x) + m*np.log(np.exp(x) - 1)
    A = phi(x0) - m*t0 + (n-m)*np.log(t0)
    # typically f() will be called w/argument t0
    def f(x):
        return sqrt(x / ((1+x) * (x0-x)) )
    # needed for second order approx
    def f1(x):
        num = -2*x0**3 + 2*x**5 + 4*x**4 + 4*x**3 + 3*x*x0**2 - 6*x0*x**4 - 5*x0**2*x**2 + 2*x0**4*x + x0**3*x - 6*x0**3*x**2 + 8*x0**2*x**3
        denom = 24*f(x)*(1+x)**2*(x0-x)**4
        return num/denom
    # first order approx gives
    first = np.exp(A)*m**(n-m)*f(t0)*comb(n,m)
    # second order approx gives
    second = np.exp(A)*m**(n-m)*(f(t0)-f1(t0)/m)*comb(n,m)
    rel_err1 = (actual[m] - first) / first
    rel_err2 = (actual[m] - second) / second
    if abs(rel_err1) > m1[0]:
        m1 = (abs(rel_err1), m)
    if abs(rel_err2) > m2[0]:
        m2 = (abs(rel_err2), m)
    print(f"n={n}, m={m}, first = {first}, second={second}, actual={actual[m]}, rel_err1={rel_err1}, rel_err2={rel_err2}")

print(m1, m2)

@rlucas7 Cool. Yeah, I agree we should start with the second order approximation. Later on if we get bandwidth we can try to independently work out higher order approximations.
I also have the following approximation based on the inclusion/exclusion formula, and using the Lanczos approximation to tame the factorials, cancelling as much as possible. This can be used for smaller values.

def stirling2_double(n, k):
    result = 0
    k_choose_j = 1
    for j in range(k):
        term = (
            (-1)**j * k_choose_j *
            ((1 - j/k) / (1 + (lanczos_g + 0.5)/k) * np.e)**(k + 0.5)
            * (k - j)**(n - k - 0.5)
        )
        k_choose_j = k_choose_j * (k - j) / (j + 1)
    return result / _lanczos_sum_expg_scaled(k + 1)

Let's try to get this in during the week, and I'll submit a PR for the approximation next weekend.

It looks like the docs still aren't rendering the brace notation (and the r""" is now in the docs. Let me know if anyone has any ideas on how to fix that.

If you want to check yourself the link is:
https://output.circle-artifacts.com/output/job/17f05edd-f24b-4c0d-ad13-89a0030bf835/artifacts/0/html/reference/generated/scipy.special.stirling2.html
it looks like there are a few outstanding items on this one still-besides the docstring rendering properly.
There are also approximation method(s) to be added.
@steppi if you want I can put together what I had for Temme's second order and add it here to the existing PR?
I spent some time studying the Lanczos approaches in Pugh's thesis I'm still working through the Chebyshev approximation part but I get the gist of that particular approach. I didn't have access to the reference so I needed to work out the derivation for Lemma 3.2. In any case I feel comfortable reviewing a PR for that work if you wanted to put up something for lower values of (n,k), e.g. where n < 300. Using the inclusion-exclusion and Lanczos approximation you indicated. I haven't tested out either of the numerical approaches indicated in the thesis so I'm not sure of their strengths/weaknesses using actual values.
I'm also not sure what the consensus is on the 0-dimension numpy array, do we need to change the behavior in this PR? I think the answer is yes but want to confirm before adding another change.

I'm also not sure what the consensus is on the 0-dimension numpy array, do we need to change the behavior in this PR? I think the answer is yes but want to confirm before adding another change.

Let’s follow @h-vetinari’s precedent for the factorial functions. He made a good case for it in gh-18768.

@steppi if you want I can put together what I had for Temme's second order and add it here to the existing PR?

Let’s save it for a followup PR. Once the doc issues are resolved and the behavior lines up with the factorial functions regarding 0d arrays I think it’ll be ready to merge.

Besides  the 0d array thing, we made stirling2 accept object array of int input and the factorial functions don’t. We should probably make stirling2 match the factorial functions with respect to this too. @h-vetinari do you have any thoughts?

Besides the 0d array thing, we made stirling2 accept object array of int input and the factorial functions don’t. We should probably make stirling2 match the factorial functions with respect to this too. @h-vetinari do you have any thoughts?

The reason we used object as the dtype was for arbitrary precision integers b/c the stirling numbers of the second kind overflow quickly for both 32 and 64 bit integers. If we switched back we'd be dealing with the precision issue again and it would severely limit the usefulness of this function. E.g. all n>27 IIRC.

Besides the 0d array thing, we made stirling2 accept object array of int input and the factorial functions don’t. We should probably make stirling2 match the factorial functions with respect to this too. @h-vetinari do you have any thoughts?

The reason we used object as the dtype was for arbitrary precision integers b/c the stirling numbers of the second kind overflow quickly for both 32 and 64 bit integers. If we switched back we'd be dealing with the precision issue again and it would severely limit the usefulness of this function. E.g. all n>27 IIRC.

Sorry. That’s not what I mean. The factorial functions also work with Python ints internally and return an object array, but the input array must be some kind of numpy int dtype.

Besides the 0d array thing, we made stirling2 accept object array of int input and the factorial functions don’t. We should probably make stirling2 match the factorial functions with respect to this too. @h-vetinari do you have any thoughts?

The reason we used object as the dtype was for arbitrary precision integers b/c the stirling numbers of the second kind overflow quickly for both 32 and 64 bit integers. If we switched back we'd be dealing with the precision issue again and it would severely limit the usefulness of this function. E.g. all n>27 IIRC.

Sorry. That’s not what I mean. The factorial functions also work with Python ints internally and return an object array, but the input array must be some kind of numpy int dtype. I’d made a suggestion earlier to allow object array input which may actually be bad from a consistency standpoint.

Besides the 0d array thing, we made stirling2 accept object array of int input and the factorial functions don’t. We should probably make stirling2 match the factorial functions with respect to this too. @h-vetinari do you have any thoughts?

The reason we used object as the dtype was for arbitrary precision integers b/c the stirling numbers of the second kind overflow quickly for both 32 and 64 bit integers. If we switched back we'd be dealing with the precision issue again and it would severely limit the usefulness of this function. E.g. all n>27 IIRC.

Sorry. That’s not what I mean. The factorial functions also work with Python ints internally and return an object array, but the input array must be some kind of numpy int dtype. I’d made a suggestion earlier that would let stirling2 accept object arrays as input which might have been bad from a consistency standpoint.

Ah, I see what you mean. IIRC the current implementation raises an error in the case when the input is an object array in the input. There is a test that confirms this behavior in this PR. Is there something that needs to be done for the object input beyond what's already done?

@steppi SGTM would you like me to put together a separate PR for the Temme approximation?

Yes, that would be awesome.


[DNM] feat: Use new LaunchPadState

Description
This PR introduces changes to cater for the new LaunchPadState structure - currently only being used in the Dashboard ExpertView.
Changes

Added a hook to simply listen for deltas
Added store for launchpadState, including:

new types
action to set the State from LaunchPadState once off
action to send action (using emit('tari://actions', { Action: { ChangeSession: sessionUpdate } }))
push any Deltas into a redux state array that can be used where needed (this should be used in the selectors, but the types are all very different - started implementation in shanimal_implement/lps, but unsure about where to find container stats in the new structure)

Small typo fix & addressed some console warnings

Outstanding (before i would consider this merge-able)

[ ] NB - Container Stats data - Expert view is not great without all that information

started these changes in shanimal_implement/lps, but the BE needs to be updated

[ ] Implementing the changes in other sections.

Motivation and Context
This is the start of trying to have the FE use the new structure for LaunchPadState - more info in the readme
How Has This Been Tested?
Manually, locally. Mostly with a lot of console.logs :) but you can now use the star/stop buttons in expert view

Obsolete


check doens't filter named errors

Description:
        io:ReadableCSVChannel rCsvChannel = checkpanic io:openReadableCsvFile(filePath);
        if (rCsvChannel.hasNext()) {
            string[] titles = check rCsvChannel.getNext();
        }

incompatible types: expected '(string[]|error)', found '(string[]|ballerina/io:GenericError|ballerina/io:ConnectionTimedOutError)?'

Steps to reproduce:
Affected Versions:
Ballerina 1.0.0-beta-SNAPSHOT
OS, DB, other environment details and versions:
Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

This is a bug in the source code, albeit error message is difficult to decipher.
string[]? titles = check rCsvChannel.getNext();


Equivalent of simlayer from Goku

Hi,
I just found your library and it looks quite good. Especially because I find it hard to get my head around the DSL from Goku and wanted to give your idea a try.
What I'm searching for is the equivalent of a simlayer from goku (see: https://github.com/yqrashawn/GokuRakuJoudo/blob/master/tutorial.md#advance3).
What I would like to do is to have a layer that is activating on "holding down f".
What I did until now is this:
import {
    HyperKey,
    KarabinerComplexModifications,
} from<EMAIL_ADDRESS>
const f_mode = new HyperKey({
    id: "f_mode",
    description: "F Mode",
    from: {
        key_code: "f"
    }
});

f_mode.bindKey({symbol: "j", description: "j", key: "j", to: [{key_code: "left_arrow"}]});
f_mode.bindKey({symbol: "k", description: "k", key: "k", to: [{key_code: "down_arrow"}]});
f_mode.bindKey({symbol: "i", description: "i", key: "i", to: [{key_code: "up_arrow"}]});
f_mode.bindKey({symbol: "l", description: "l", key: "l", to: [{key_code: "right_arrow"}]});

const mods = new KarabinerComplexModifications();

mods.addRule(f_mode.getRules());
mods.writeToProfile("Deno");

When I hold down f it works (using jkil for navigation), but when I only tab f it does not print the regular f character.
Would you be able to take a look and help me out?
Cheers
Mario

You can do it with to_if_alone and to_if_held_down but unfortunately the HyperKey abstraction does not support this (for now at least). But you can certainly do it with custom mods.addRule() calls.
I did something like this for mute
https://github.com/esamattis/deno_karabiner/blob/b30984f6daf803277c0444d760c68ba12b2e9c6c/example.ts#L710-L742


JDK Wrapper fails cryptically when not setting JDKW_DIST

When not setting "JDKW_DIST" the wrapper fails with a cryptic error:
[01:09:48] ERROR: curl  -f -k -L -o "/var/tmp/builder.cache/jdk/jdkw//jdkw-impl.sh" "https://github.com/KoskiLabs/jdk-wrapper/releases/download//jdkw-impl.sh" failed with 127
It would be nice to explicitly check for absence instead and error out before attempting the curl.

Yes, the error handling needs to be improved. I’ll take care of that. Do you know why it failed to detect the version? Do you have the complete verbose output?

So I've been unable to reproduce the error with no JDKW_DIST specified; instead I get this:
> ./jdk-wrapper.sh JDKW_DIST= java -version
[18:18:28] Unsupported distribution

The error you printed is coming from jdk-wrapper.sh trying to download the implementation (jdkw-impl.sh). It happens when the script failed to determine the latest release (or have JDKW_RELEASE) set. What value do you have for JDKW_RELEASE?
If you are using latest (or default) for JDKW_RELEASE then the script curls Github to determine the latest release before invoking that release. Did invoke the script without internet connectivity?
I'm adding a better warning if the latest fails to resolve, which I think is the case here because if the // in the URI. However, I'd like to understand why it failed to resolve. If you could reply with answer to the two questions above as well as any additional output prior to the error, that would be helpful.

I've just tried to reproduce this and was no longer able to with the newest version. I also verified that everything works as expected on Darwin. Thanks for following up so promptly.


[BUG] When deleting the cluster in autok3s, the data directory defined by --data-dir was not deleted

mount |grep kubelet
tmpfs on /work/lib/rancher/k3s/kubelet/pods/fc3e5ab8-b3d3-417e-b605-8c5532c69011/volumes/kubernetes.io~secret/config type tmpfs (rw,relatime)
tmpfs on /work/lib/rancher/k3s/kubelet/pods/fc3e5ab8-b3d3-417e-b605-8c5532c69011/volumes/kubernetes.io~secret/js-loki-stack-promtail-token-9fz8v type tmpfs (rw,relatime)

Seems like you are not using the default data-dir to run k3s. In the k3s-uninstall.sh script, it only do unmount to the default data-dir path.
Because the data-dir parameter can be changed in runtime, k3s uninstall won't respect to the current configuration and just unmount the default path.
So I don't think this is a bug for autok3s. You can raise an issue in https://github.com/k3s-io/k3s for more information.

Feel free to re-open this for further discussion.


Dependency order

Find Cabana before shared dependent libraries - I can build with only the Cabana install path

Codecov Report

Merging #40 (b55530f) into master (54ad47a) will increase coverage by 0.44%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #40      +/-   ##
==========================================
+ Coverage   93.02%   93.46%   +0.44%
==========================================
  Files          43       41       -2
  Lines        4129     2878    -1251
==========================================
- Hits         3841     2690    -1151
+ Misses        288      188     -100

Impacted Files
Coverage Δ

src/Picasso_ParticleList.hpp
60.00% <0.00%> (-16.93%)
:arrow_down:

unit_test/tstLevelSetRedistance.hpp
87.09% <0.00%> (-12.91%)
:arrow_down:

src/Picasso_UniformMesh.hpp
69.38% <0.00%> (-10.62%)
:arrow_down:

unit_test/tstParticleInit.hpp
93.93% <0.00%> (-6.07%)
:arrow_down:

src/Picasso_InputParser.cpp
89.47% <0.00%> (-2.84%)
:arrow_down:

src/Picasso_AdaptiveMesh.hpp
83.33% <0.00%> (-2.39%)
:arrow_down:

src/Picasso_UniformCartesianMeshMapping.hpp
96.55% <0.00%> (-1.13%)
:arrow_down:

src/Picasso_FieldManager.hpp
97.43% <0.00%> (-0.68%)
:arrow_down:

src/Picasso_BatchedLinearAlgebra.hpp
99.46% <0.00%> (-0.54%)
:arrow_down:

unit_test/tstBatchedLinearAlgebra.hpp
97.04% <0.00%> (-0.46%)
:arrow_down:

... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54ad47a...b55530f. Read the comment docs.

I don't get what the issue was and how it is being resolved.

Were you getting HYPRE errors?

Were you getting HYPRE errors?

I've never had any issues building with hypre, but I figured we should fix whatever your issues are in a separate PR

I don't get what the issue was and how it is being resolved.

I can't reproduce my previous issue, so this PR can be closed if it doesn't fix Stuart's issue


feat: use helm as a libary instead of copying code

This change would include the official helm code as a dependency instead of copying only the code used for linting etc. to this project.
I would really like to get your input @mrjosh on such a design decision. Other people are also welcome to the discussion.
Advantages:

Less code in this repo that is hard to maintain
New lints would be added
We could include more features interacting with helm directly in the future

Disadvantages:

Larger binary size (went from 49M to 74M) without compression and to 29M using upx
Possible breaking changes when updating (adding enough tests should avoid this)

This would revert https://github.com/mrjosh/helm-ls/pull/14

Using helm as a library does not seem uncommon, quite a lot of projects do this: https://github.com/helm/helm/network/dependents?package_id=UGFja2FnZS0yMjY3Mjg4NjYw

Hi @qvalentin thanks for the info regarding using helm-core as library.
By leveraging the Helm code directly, we inherently reduce the overhead associated with maintaining a separate, potentially divergent codebase. This shift not only streamlines our development process but also ensures that we benefit from the latest improvements and features added to Helm, keeping our project up-to-date and robust.
Regarding the concerns about the increased binary size and potential breaking changes, these are valid points. However, the reduction in binary size with compression to 29M is quite efficient, and by prioritizing thorough testing, we can mitigate the risks of breaking changes effectively. This proactive approach in testing will safeguard our project's stability and reliability.

Thanks for your opinion, I will take this as a approve for the proposed changes and will make it ready to be merged soon.


Add license & update all dependencies

Hi,
would you be open to add an Open Source license for the project?
Here is a list https://api.github.com/licenses
I always tend to MIT.
Second question => should we update the dependencies? i got a vulnerability in node-fetch.
I already tried a complete update with
ncu -u command and it seemed to work well. Would have to check closer.
Best regards

Hey 👋🏻,
Updated license.
If needed you can update the dependencies. Feel free to make a pull request.
Thank you

Alright,
I'm working on the responsiveness & updates.


feat: recursively resolve input

See also: Resolvers documentation
This adds a recursive implementation for the input resolver. Before this change we were only resolving the top-level input struct and any embedded anonymous structs. This missing feature that was highlighted while trying to handle custom validation for Protobuf → Huma specifically around proto one-of fields in deeply nested input data structures. It does two important things:

Walk the entire input structure looking for resolvable structs (including arrays, maps, structs)
Generate a correct JSON Path when nested resolvers add errors

The included test exercises both of the above and I've confirmed this is working with protoc-gen-huma locally.

Codecov Report

Merging #16 (e9e9d97) into main (f77c2b1) will increase coverage by 0.58%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #16      +/-   ##
==========================================
+ Coverage   82.72%   83.30%   +0.58%
==========================================
  Files          20       20
  Lines        1308     1342      +34
==========================================
+ Hits         1082     1118      +36
+ Misses        167      166       -1
+ Partials       59       58       -1

Impacted Files
Coverage Δ

operation.go
74.03% <100.00%> (+0.25%)
:arrow_up:

resolver.go
82.27% <100.00%> (+3.84%)
:arrow_up:

schema/schema.go
92.85% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f77c2b1...f08e6b7. Read the comment docs.


Getting data from ESGF @bouweandela

Dear Bouwe,
Following our discussions at the ESMValTool workshop I used the below python commands to get data from the ESGF.
Hovewer I stopped the transfer as I got twice the file
ch4_Amon_BCC-CSM2-MR_amip_r1i1p1f1_gn_197901-201412.nc
I got on my machine:
ch4_Amon_BCC-CSM2-MR_amip_r1i1p1f1_gn_197901-201412.nc.9m0tlfp5
and
ch4_Amon_BCC-CSM2-MR_amip_r1i1p1f1_gn_197901-201412.nc.5ir7x0qd
So I stopped the transfer in case the file, and others, would be loaded several times each...
Any suggestion on this,
Thanks for your response.
All the best,
Martine

import yaml
from esmvalcore.config import CFG
from esmvalcore.dataset import Dataset
from esmvalcore.esgf import download
CFG["search_esgf"] = "always"
dataset_template = Dataset(
short_name="ch4",
mip="Amon",
project="CMIP6",
dataset="",
ensemble="",
institute="",
exp="amip",
grid="",
)
datasets = list(dataset_template.from_files())
dataset = datasets[0]
dataset.files[0].download(CFG["download_dir"])

Ignore this issue thank you.


chore: remove datataset from title

SUMMARY
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

Merging #12344 (a8bc297) into master (f482849) will decrease coverage by 16.30%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #12344       +/-   ##
===========================================
- Coverage   67.15%   50.84%   -16.31%
===========================================
  Files        1002      464      -538
  Lines       49324    16836    -32488
  Branches     5020     4329      -691
===========================================
- Hits        33124     8561    -24563
+ Misses      16075     8275     -7800
+ Partials      125        0      -125

Flag
Coverage Δ

cypress
50.84% <ø> (-0.66%)
:arrow_down:

javascript
?

python
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...et-frontend/src/explore/controlPanels/sections.jsx
100.00% <ø> (ø)

...uperset-frontend/src/dashboard/util/dnd-reorder.js
0.00% <0.00%> (-100.00%)
:arrow_down:

...rset-frontend/src/dashboard/util/getEmptyLayout.js
0.00% <0.00%> (-100.00%)
:arrow_down:

...dashboard/components/resizable/ResizableHandle.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/dashboard/util/getFilterScopeFromNodesTree.js
0.00% <0.00%> (-93.48%)
:arrow_down:

...set-frontend/src/views/CRUD/alert/ExecutionLog.tsx
11.76% <0.00%> (-88.24%)
:arrow_down:

...src/dashboard/components/gridComponents/Header.jsx
10.52% <0.00%> (-86.85%)
:arrow_down:

superset-frontend/src/components/IconTooltip.tsx
13.33% <0.00%> (-86.67%)
:arrow_down:

...rc/dashboard/components/gridComponents/Divider.jsx
13.33% <0.00%> (-86.67%)
:arrow_down:

...end/src/SqlLab/components/ExploreResultsButton.jsx
8.00% <0.00%> (-84.00%)
:arrow_down:

... and 861 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f482849...a8bc297. Read the comment docs.

Codecov Report

Merging #12344 (a8bc297) into master (f482849) will decrease coverage by 16.30%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #12344       +/-   ##
===========================================
- Coverage   67.15%   50.84%   -16.31%
===========================================
  Files        1002      464      -538
  Lines       49324    16836    -32488
  Branches     5020     4329      -691
===========================================
- Hits        33124     8561    -24563
+ Misses      16075     8275     -7800
+ Partials      125        0      -125

Flag
Coverage Δ

cypress
50.84% <ø> (-0.66%)
:arrow_down:

javascript
?

python
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...et-frontend/src/explore/controlPanels/sections.jsx
100.00% <ø> (ø)

...uperset-frontend/src/dashboard/util/dnd-reorder.js
0.00% <0.00%> (-100.00%)
:arrow_down:

...rset-frontend/src/dashboard/util/getEmptyLayout.js
0.00% <0.00%> (-100.00%)
:arrow_down:

...dashboard/components/resizable/ResizableHandle.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/dashboard/util/getFilterScopeFromNodesTree.js
0.00% <0.00%> (-93.48%)
:arrow_down:

...set-frontend/src/views/CRUD/alert/ExecutionLog.tsx
11.76% <0.00%> (-88.24%)
:arrow_down:

...src/dashboard/components/gridComponents/Header.jsx
10.52% <0.00%> (-86.85%)
:arrow_down:

superset-frontend/src/components/IconTooltip.tsx
13.33% <0.00%> (-86.67%)
:arrow_down:

...rc/dashboard/components/gridComponents/Divider.jsx
13.33% <0.00%> (-86.67%)
:arrow_down:

...end/src/SqlLab/components/ExploreResultsButton.jsx
8.00% <0.00%> (-84.00%)
:arrow_down:

... and 861 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f482849...a8bc297. Read the comment docs.

LGTM

LGTM


Remove "EdgeML"

Resolves https://github.com/arduino/library-registry/issues/936

Backend work done:
2022/01/26 14:53:39 Loaded 4580 libraries from DB
Removing EdgeML
Success!


Remove click pin

The latest version of click addresses the typing issues of click.group() meaning we should be able to unpin it now.

The best way I can see is explicitly going,
click!=8.1.0,!=8.1.1,!=8.1.2
etc until 8.1.4. It seems our issues are fixed only in 8.1.5. This seems messy to me but I'm happy to implement it.

That sounds like an overkill, it's only the type checking that has problems, right?

Should be able to do click>=8.1.5,<8.1.0. See PEP508

I thought everything up to and including 8.1.3 worked, and just 8,1,4 and 8.1.5 didn't?


gateway / implement node/databases api

Move api from backend (centralized) to gateway (decentralized)
documentation:
https://github.com/sqlitecloud/backend/blob/main/sqliteweb/doc/dashboard/v1/{projectID}/databases/GET.md
implementation:
https://github.com/sqlitecloud/backend/blob/a12cf5308c4eb09d90ec2a8f5a4f87d3200f477f/sqliteweb/dashboard/v1/{projectID}/databases/GET.lua#L4

Seems like the output of the API is for the most part the same as a:
LIST DATABASES DETAILED
except for a couple changes in stats and bytes which already caused problems in dashboard (because they were inverted).
I will be sending the output as it, so api can later be replaced with a direct call to DB if wish be.


[ci] - fix slack hook for ci

fixing the slack hook value for vault

i just realized that the slack hook PROBABLY isn't needed currently due to the slack integration being removed in the slack channel
removed an integration from this channel: Duty Free Automation

down the road we might wanna use it to show failing tests/builds?

i'm gonna go ahead and close this cause its probably not relevant anymore


Passing market depth data and spread to the client

How would the code need to be extended in order to pass market depth / order book data to the client?
Which MQL functions should be called to fetch this data from MT?
And which MQL functions should be called to get the spread for a symbol?

Hi,
Regarding the spread, I would not add a new function for that. If you are subscribed to a symbol, you can just use "ask minus bid" on the python side to get the spread.
Market depth would only work on MT5, not MT4. DWX Connect does not offer this function yet, so you would have to add it yourself.
Here is a great video from Martyn explaining how to get market depth data:
An here is a link to the mql documentation.


dev1.0.0: Folder tree and executable bash script

Release dev1.0.0:

Created folder tree
Added create_requirements.sh
Added installation guide in DEPENDENCIES.md

Merged Ok.


Python package version bumps

to:
cc: @airbnb/streamalert-maintainers
related to:
resolves:
Background
This PR resolves 2 errors i encountered when deploying streamalert on Macos Big Sur with Python3.7.
The first is when running pip install -r requirements.txt
Collecting cffi==1.14.1
  Using cached cffi-1.14.1.tar.gz (468 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/dbi/Code/streamalert/venv/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/pq/pd76h5710d5b0f4nbbf_dvzc0000gn/T/pip-install-0t13an5f/cffi/setup.py'"'"'; __file__='"'"'/private/var/folders/pq/pd76h5710d5b0f4nbbf_dvzc0000gn/T/pip-install-0t13an5f/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/pq/pd76h5710d5b0f4nbbf_dvzc0000gn/T/pip-pip-egg-info-4y77wv44
         cwd: /private/var/folders/pq/pd76h5710d5b0f4nbbf_dvzc0000gn/T/pip-install-0t13an5f/cffi/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/pq/pd76h5710d5b0f4nbbf_dvzc0000gn/T/pip-install-0t13an5f/cffi/setup.py", line 152, in <module>
        if 'darwin' in sys.platform and macosx_deployment_target() >= (10, 15):
      File "/private/var/folders/pq/pd76h5710d5b0f4nbbf_dvzc0000gn/T/pip-install-0t13an5f/cffi/setup.py", line 76, in macosx_deployment_target
        return tuple(map(int, get_config_var("MACOSX_DEPLOYMENT_TARGET").split('.')))
    AttributeError: 'int' object has no attribute 'split'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The second is when building the lambda functions
[INFO 2021-02-08 17:55:51,707 (streamalert_cli.manage_lambda.package:66)]: Creating package for streamalert
[INFO 2021-02-08 17:55:51,925 (streamalert_cli.manage_lambda.package:124)]: Installing libraries: boto3==1.14.29, jmespath==0.10.0, netaddr==0.8.0, backoff==1.10.0, pymsteams==0.1.13, policyuniverse==<IP_ADDRESS>, pathlib2==2.3.5, jsonlines==1.2.0, google-api-python-client==1.10.0, requests==2.24.0, cbapi==1.7.1
ERROR: moto 1.3.14 has requirement idna<2.9,>=2.5, but you'll have idna 2.10 which is incompatible.

Changes

Update cffi to 1.14.4
Explicitly install idna 2.8 to prevent install error in moto 1.3.14

Testing
I have ran the unit tests, however i'm new to streamalert so have not performed any kind of detailed testing.

Rebasing onto release-3-5-0


SynAnt closes itself when using cmd+enter/alt+enter

Setup info

Workflow name: SynAnt
Workflow version: 21.1
Alfred version: 4.6.2
macOS version: 11.6.2
Architecture: Intel

Description of issue
When I try to cmd+enter to search for the synonyms of an list item, it closes the alfred window and can't see the results.
I can fix this by enabling "Don't close the Alfred Window on actioning result" checkbox, but I think it should be the default.

I can’t reproduce the problem; it always worked fine come me because it opens the window right away. I’d be interested in seeing a video of the behaviour with the debugger open and visible.
But activating this doesn’t make a difference to me, so if it fixes it for you, I have indeed made it the default (just released).


⚠ Bofrost.lu has degraded performance

In 75fab91, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1043 ms

Resolved: Bofrost.lu performance has improved in 927cb90 after 7 minutes.


Allow for download of pipe run log files from the web console

Currently users have to access the Cloudant web console to download data pipe run log files, which is a major inconvenience.

Deliverd by #49