Add NeonDB

https://github.com/tyaga001/awesome-neondb
Neon is designed for a wide range of users, from individual devs to enterprises, seeking modern, serverless Postgres capabilities. It caters to those who need a fully managed, scalable, and cost-effective database solution.]
My comments for the pull requests to be reviewed:
https://github.com/sindresorhus/awesome/pull/3106#issuecomment-2211825969
https://github.com/sindresorhus/awesome/pull/3068#issuecomment-2206817069
https://github.com/sindresorhus/awesome/pull/3092#issuecomment-2208974372
By submitting this pull request I confirm I've read and complied with the below requirements 🖖
Please read it multiple times. I spent a lot of time on these guidelines and most people miss a lot.
Requirements for your pull request

[x] Don't open a Draft / WIP pull request while you work on the guidelines. A pull request should be 100% ready and should adhere to all the guidelines when you open it. Instead use #2242 for incubation visibility.
[x] Don't waste my time. Do a good job, adhere to all the guidelines, and be responsive.
[x] You have to review at least 2 other open pull requests.
Try to prioritize unreviewed PRs, but you can also add more comments to reviewed PRs. Go through the below list when reviewing. This requirement is meant to help make the Awesome project self-sustaining. Comment here which PRs you reviewed. You're expected to put a good effort into this and to be thorough. Look at previous PR reviews for inspiration. Just commenting “looks good” or simply marking the pull request as approved does not count! You have to actually point out mistakes or improvement suggestions. Comments pointing out lint violation are allowed, but does not count as a review.
[x] You have read and understood the instructions for creating a list.
[x] This pull request has a title in the format Add Name of List. It should not contain the word Awesome.

✅ Add Swift
✅ Add Software Architecture
❌ Update readme.md
❌ Add Awesome Swift
❌ Add swift
❌ add Swift
❌ Adding Swift
❌ Added Swift

[x] Your entry here should include a short description of the project/theme of the list. It should not describe the list itself. The first character should be uppercase and the description should end in a dot. It should be an objective description and not a tagline or marketing blurb. It should not contain the name of the list.

✅ - [iOS](…) - Mobile operating system for Apple phones and tablets.
✅ - [Framer](…) - Prototyping interactive UI designs.
❌ - [iOS](…) - Resources and tools for iOS development.
❌ - [Framer](…)
❌ - [Framer](…) - prototyping interactive UI designs

[x] Your entry should be added at the bottom of the appropriate category.
[x] The title of your entry should be title-cased and the URL to your list should end in #readme.

Example: - [Software Architecture](https://github.com/simskij/awesome-software-architecture#readme) - The discipline of designing and building software.

[x] No blockchain-related lists.
[x] The suggested Awesome list complies with the below requirements.

Requirements for your Awesome list

[x] Has been around for at least 30 days.That means 30 days from either the first real commit or when it was open-sourced. Whatever is most recent.
[x] Run awesome-lint on your list and fix the reported issues. If there are false-positives or things that cannot/shouldn't be fixed, please report it.
[x] The default branch should be named main, not master.
[x] Includes a succinct description of the project/theme at the top of the readme. (Example)

✅ Mobile operating system for Apple phones and tablets.
✅ Prototyping interactive UI designs.
❌ Resources and tools for iOS development.
❌ Awesome Framer packages and tools.

[x] It's the result of hard work and the best I could possibly produce.
If you have not put in considerable effort into your list, your pull request will be immediately closed.
[x] The repo name of your list should be in lowercase slug format: awesome-name-of-list.

✅ awesome-swift
✅ awesome-web-typography
❌ awesome-Swift
❌ AwesomeWebTypography

[x] The heading title of your list should be in title case format: # Awesome Name of List.

✅ # Awesome Swift
✅ # Awesome Web Typography
❌ # awesome-swift
❌ # AwesomeSwift

[x] Non-generated Markdown file in a GitHub repo.
[x] The repo should have awesome-list & awesome as GitHub topics. I encourage you to add more relevant topics.
[x] Not a duplicate. Please search for existing submissions.
[x] Only has awesome items. Awesome lists are curations of the best, not everything.
[x] Does not contain items that are unmaintained, has archived repo, deprecated, or missing docs. If you really need to include such items, they should be in a separate Markdown file.
[x] Includes a project logo/illustration whenever possible.

Either centered, fullwidth, or placed at the top-right of the readme. (Example)
The image should link to the project website or any relevant website.
The image should be high-DPI. Set it to a maximum of half the width of the original image.
Don't include both a title saying Awesome X and a logo with Awesome X. You can put the header image in a # (Markdown header) or <h1>.

[x] Entries have a description, unless the title is descriptive enough by itself. It rarely is though.
[x] Includes the Awesome badge.

Should be placed on the right side of the readme heading.

Can be placed centered if the list has a centered graphics header.

Should link back to this list.

[x] Has a Table of Contents section.

Should be named Contents, not Table of Contents.
Should be the first section in the list.
Should only have one level of nested lists, preferably none.
Must not feature Contributing or Footnotes sections.

[x] Has an appropriate license.

We strongly recommend the CC0 license, but any Creative Commons license will work.

Tip: You can quickly add it to your repo by going to this URL: https://github.com/<user>/<repo>/community/license/new?branch=main&template=cc0-1.0 (replace <user> and <repo> accordingly).

A code license like MIT, BSD, Apache, GPL, etc, is not acceptable. Neither are WTFPL and Unlicense.
Place a file named license or LICENSE in the repo root with the license text.
Do not add the license name, text, or a Licence section to the readme. GitHub already shows the license name and link to the full text at the top of the repo.
To verify that you've read all the guidelines, please comment on your pull request with just the word unicorn.

[x] Has contribution guidelines.

The file should be named contributing.md. The casing is up to you.
It can optionally be linked from the readme in a dedicated section titled Contributing, positioned at the top or bottom of the main content.
The section should not appear in the Table of Contents.

[x] All non-important but necessary content (like extra copyright notices, hyperlinks to sources, pointers to expansive content, etc) should be grouped in a Footnotes section at the bottom of the readme. The section should not be present in the Table of Contents.
[x] Has consistent formatting and proper spelling/grammar.

The link and description are separated by a dash. Example: - [AVA](…) - JavaScript test runner.
The description starts with an uppercase character and ends with a period.
Consistent and correct naming. For example, Node.js, not NodeJS or node.js.

[x] Does not use hard-wrapping.
[x] Does not include a CI (e.g. GitHub Actions) badge.You can still use a CI for linting, but the badge has no value in the readme.
[x] Does not include an Inspired by awesome-foo or Inspired by the Awesome project kinda link at the top of the readme. The Awesome badge is enough.

Go to the top and read it again.

Thanks for making an Awesome list! 🙌
It looks like you didn't read the guidelines closely enough. I noticed multiple things that are not followed. Try going through the list point for point to ensure you follow it. I spent a lot of time creating the guidelines so I wouldn't have to comment on common mistakes, and rather spend my time improving Awesome.

I double checked everything and looks ok could you please check now I'm only getting this which seems false positive.

unicorn

Thanks for making an Awesome list! 🙌
It looks like you didn't read the guidelines closely enough. I noticed multiple things that are not followed. Try going through the list point for point to ensure you follow it. I spent a lot of time creating the guidelines so I wouldn't have to comment on common mistakes, and rather spend my time improving Awesome.

I double checked everything and looks ok could you please check now I'm only getting this which seems false positive.

Hello, you have not checked everything closely enough. If you go to the actions section you will see what is failing for lint for your repository.
README.md:5:1
  ✖    1:1   Main heading must be in title case                                           remark-lint:awesome-heading
  ✖    1:1   The repository should have "awesome" as a GitHub topic                       remark-lint:awesome-github
  ✖    1:1   The repository should have "awesome-list" as a GitHub topic                  remark-lint:awesome-github
  ✖    1:1   Forbidden license section found                                              remark-lint:awesome-license
  ✖    3:1   https://neon.tech                                                            remark-lint:double-link
  ✖    5:1   Unexpected character : at end of heading, remove it                          remark-lint:no-heading-punctuation
  ✖    5:1   Unexpected duplicate toplevel heading, exected a single heading with rank 1  remark-lint:no-multiple-toplevel-headings
  ✖    5:1   List can only have one heading                                               remark-lint:awesome-heading
  ✖   [17](https://github.com/sindresorhus/awesome/actions/runs/9780138353/job/27001062448?pr=3103#step:3:18):43  CONTRIBUTING.md                                                              remark-lint:double-link
  ✖   19:1   Table of Contents must be the first section                                  remark-lint:awesome-toc
  ✖   20:1   ToC missing item for "License"                                               remark-lint:awesome-toc
  ✖   31:3   https://neon.tech/                                                            remark-lint:double-link
  ✖   62:39  "’" is used without matching "‘"                                             remark-lint:match-punctuation
  ✖  110:13  CONTRIBUTING.md                                                              remark-lint:double-link

Additionally I do not see a single link that has a description of what it is about.
I see that you have a License file but it looks like you just copied and pasted the legal code into a file titled "License". Additionally the legal code is not complete as it cuts off and attempts a markdown - hyperlink to the site to let readers read the rest which is not convenient.

Thanks for making an Awesome list! 🙌
It looks like you didn't read the guidelines closely enough. I noticed multiple things that are not followed. Try going through the list point for point to ensure you follow it. I spent a lot of time creating the guidelines so I wouldn't have to comment on common mistakes, and rather spend my time improving Awesome.

I double checked everything and looks ok could you please check now I'm only getting this which seems false positive.

Hello, you have not checked everything closely enough. If you go to the actions section you will see what is failing for lint for your repository. I run in my local and it only give this error which is false positive

README.md:5:1
  ✖    1:1   Main heading must be in title case                                           remark-lint:awesome-heading
  ✖    1:1   The repository should have "awesome" as a GitHub topic                       remark-lint:awesome-github
  ✖    1:1   The repository should have "awesome-list" as a GitHub topic                  remark-lint:awesome-github
  ✖    1:1   Forbidden license section found                                              remark-lint:awesome-license
  ✖    3:1   https://neon.tech                                                            remark-lint:double-link
  ✖    5:1   Unexpected character : at end of heading, remove it                          remark-lint:no-heading-punctuation
  ✖    5:1   Unexpected duplicate toplevel heading, exected a single heading with rank 1  remark-lint:no-multiple-toplevel-headings
  ✖    5:1   List can only have one heading                                               remark-lint:awesome-heading
  ✖   [17](https://github.com/sindresorhus/awesome/actions/runs/9780138353/job/27001062448?pr=3103#step:3:18):43  CONTRIBUTING.md                                                              remark-lint:double-link
  ✖   19:1   Table of Contents must be the first section                                  remark-lint:awesome-toc
  ✖   20:1   ToC missing item for "License"                                               remark-lint:awesome-toc
  ✖   31:3   https://neon.tech/                                                            remark-lint:double-link
  ✖   62:39  "’" is used without matching "‘"                                             remark-lint:match-punctuation
  ✖  110:13  CONTRIBUTING.md                                                              remark-lint:double-link

Additionally I do not see a single link that has a description of what it is about.- all links updated
I see that you have a License file but it looks like you just copied and pasted the legal code into a file titled "License". - Additionally the legal code is not complete as it cuts off and attempts a markdown - hyperlink to the site to let readers read the rest which is not convenient.- updated
Comments pointing out lint violation are allowed, but does not count as a review.? - pls help as most PR already reviewed.
And additionally the entry should not include the name of the list. (The word Neon at the beginning of the entry.)- updated

Wise is the best bank I have found to get paid quicky and with the best conversion rates. ...
Once I have transferred once, it becomes just few clicks to transfer money between bank accounts internationally. ...
A good option (and better alternate than Payoneer) for transferring money/payments
Get Connected Now
https://wise.com/invite/ahpc/nelsong667

unicorn

Comments pointing out lint violation are allowed, but does not count as a review.

Comments pointing out lint violation are allowed, but does not count as a review.

My comments for the pull requests to be reviewed:
https://github.com/sindresorhus/awesome/pull/3106#issuecomment-2211825969
https://github.com/sindresorhus/awesome/pull/3068#issuecomment-2206817069
https://github.com/sindresorhus/awesome/pull/3092#issuecomment-2208974372

@sindresorhus can we merge it now pls let me know what I should do? I updated my PR review comments.

Consistent and correct naming. For example, Node.js, not NodeJS or node.js.

You are using both Neon, NeonDB, and Neon Postgres Database

Hi @tyaga001 :wave: ,
Great work on your list! Here a couple of small comments:

Have you considered adding the logo to your list? It seems that NeonDB does indeed have one: https://avatars.githubusercontent.com/u/77690634?s=200&v=4

You have a section called Youtube Channel but it seems to be each entry is a single video from a different channel. It's not clear to me how that section differs from Community video tutorials above, maybe the two could be merged? Alternatively, that section could point to each youtube channel of the video, if the channel has mainly neondb - related content. In this case, the section should probably be channels in plural.

Consistent and correct naming. For example, Node.js, not NodeJS or node.js.

You are using both Neon, NeonDB, and Neon Postgres Database

Updated everywhere to "Neon Postgres Database"  @sindresorhus please check if we can merge it now.

The official project name is Neon. You should stick to that.

The official project name is Neon. You should stick to that.

Updated back to Neon

You haven't updated this pull request.

@sindresorhus sorry I missed it completely, now it's done and I think it's ready to merge now. please check.

Tweet: https://x.com/awesome__re/status/1815361364665172360


proposal: cmd/gofmt: Support rare or single-purpose formatting exceptions

Proposal Details
There are times when it would be nice to format code in a custom manner and have gofmt ignore that block of code. I recognize that adding this in a naive way would pretty much defeat the purpose of gofmt. I don't want that so I'm not sure how to do what I want.
Let me try to explain with an example. I'm writing a test function that uses two different but similar structs. They have corresponding fields but due to requirements may not use the same types. Additionally, most of the fields are pointers for one reason or another.
Suppose I have (actual structs are much bigger):
type Input struct{
   AAA *SpecialInt     `db:"AAA" json:"aaa" api-required="true"`
   BB  *SpecialString  `db:"BB" json:"bb"`
   C   *SpecialString2 `db:"C" json:"c" api-required="true"`
}

type Result struct{
   A   *int    `db:"A" json:"a" api-required="true"`
   BB  *string `db:"BB" json:"bb"`
   CCC *string `db:"CCC" json:"ccc" api-required="true"`
}

Suppose in my test I have three objects:
var original Result
var input Input
var want Result

Because the fields a pointers I need to create variables to point to:
iAAA, oA, wA := SpecialInt(100), 200, 100
iBB, oBB, wBB := SpecialString("My mom says I'm special"), "nobody is special", "My mom says I'm special"
iC, oCCC, wCCC := SpecialString2("me too"), "everybody is special here", "me too"

These values are then mixed and matched as pointers in the three test objects for the various test cases. I'd like to format the preceding code block to be a bit easier to read:
iAAA, oA,   wA   := SpecialInt(100),                          200,                         100
iBB,  oBB,  wBB  := SpecialString("My mom says I'm special"), "nobody is special",         "My mom says I'm special"
iC,   oCCC, wCCC := SpecialString2("me too"),                 "everybody is special here", "me too"

It might make sense to update gofmt to format multi-assignment blocks like this or it might not. It occurs to me that there will always be special cases where special-purpose formatting would be really helpful.
Another special-purpose that gofmt doesn't currently support would be to automatically change the struct definitions above to be like this (granted since the tags are a strings, this one can be done by hand and gofmt doesn't cause any problems):
type Input struct{
    AAA *SpecialInt     `db:"AAA" json:"aaa" api-required="true"`
    BB  *SpecialString  `db:"BB"  json:"bb"`
    C   *SpecialString2 `db:"C"   json:"c"   api-required="true"`
}

type Result struct{
    A   *int    `db:"A"   json:"a"   api-required="true"`
    BB  *string `db:"BB"  json:"bb"`
    CCC *string `db:"CCC" json:"ccc" api-required="true"`
}

In the case of my examples, it might be best to implement the rules in gofmt without adding a directive that changes the behavior of gofmt within a particular block. That at least would preserve the purpose of gofmt, but it isn't flexible enough to handle all cases. Valid exceptions to the gofmt rules should be rare, but I think my example shows how supporting exceptions could be really useful. Maybe such a directive could be limited to consecutive single-line statements (no empty lines or lines with just punctuation) or it could be specifically ignored for troublesome snippets like if statements etc. I don't know what would make sense.

Thanks, but we have consistently rejected requests to customize gofmt.  In our opinion the benefits of a single canonical formatting outweigh the fact that the formatting is not ideal in every case.


Can I make draggable with some area offset constraints?

Hi,
After went through some constraints example, I know that I can add some constraints to draggable like this:
let draggable = Draggable(); runtime.add(draggable, to: imageView) {             $0.xLocked(to: 0)         }
What I want to do is make this imageView draggable within some area offset.
let's take facebook/twiiter imageViewer for an example, when user scale the image to be larger than a screen size. user will be able to pan the image left/right/up/down but the image will snap its edge to the screen when it reach the end of each edge.
Is anything like this even possible?
Thank you,

To implement that particular behavior I recommend using a zoomable UIScrollView. This will ensure that all of the system rubber banding and over-scroll behaviors are respected.

Thank you, I will check with UIScrollView


scikit-learn's Gaussian Process bug might be affecting your results

This is not necessarily an issue with bayes_opt but it's highly related and therefore I thought I should let you know about it.
Recently, I found a bug that I recently found in scikit-learn's Gaussian Process module that might be affecting the results you get from bayes_opt since bayes_opt uses scikit-learn's GP module as the surrogate model. However, in case bayes_opt only depends on the relative standard deviation values, which I think is the case, you will be safe and probably the bug won't be affecting you that much. That's why I asked the scikit-learn developers in one of the comments to add a flag whose False value will basically return the relative std values that are within [0, 1]. But in case bayes_opt depends on the actual values of the standard deviation it is almost certain that you need to run your experiments again once this bug is fixed.
Also, the scikit-learn developers say they are busy at the moment and who knows when they're gonna be able to fix the issue. So maybe you can help them fix the bug and fix this issue as soon as possible.

@fmfn Could you confirm if bayes_opt only cares about relative standard deviation values or does it actually matter for the GP module to return the actual std values?

Thanks for bringing this up! I will investigate further, but a quick glance seems to indicate it may be affecting this package.

@fmfn Sorry to ask again but I need make sure whether bayes_opt needs the actual standard deviation values or not. Could you clarify this? In case bayes_opt needs the actual std valus, do you know if there is a way to get the actual std values from the predicted std values of scikit-learn's GP that are within the range [0-1]?

@Amir-Arsalan This bug has been fixed. Would you mind closing the issue as to not confuse anyone? Thank you.


Task.Delay(timespan) imposes arbitrary bounds on timespan

Issue Title
Task.Delay(TimeSpan.FromDays(30))

throws ArgumentOutOfRangeException due to some implementation detail.
expected: APIs taking TimeSpan should support the full range of that type.

Not all APIs support the full range of their argument's values. For example, TimeSpan.FromSeconds(double) doesn't accept the full range of double values for its input. The underlying implementation imposes additional restrictions.
In your case, do you have a scenario which is blocked by Task.Delay not supporting long timeout periods (> 25-ish days)?


Scheduler and admin tweaks

Folding in older local updates to scheduler + multi-day scheduling stuff

In a rush, in the middle of the night, and at the last minute — in the great tradition of Minnebar! 😎


<optim.lbfgs> 	reached max number of iterations

It only ran for one iteration because you passed the flag num_iterations 1.

Oh my gosh I thought that was an error and didn't even check if the image came out.....

Foolish! Sorry about that

No problem - it happens to everyone from time to time ;)


clean urls?

https://github.com/seomoz/url-py

had issues building the cpython modules on my computer, just going to use unquote for now


Modules can't add pages when pages directory not exist

Environment
just stackblitz, there is no issue with environment
  System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 16.14.2 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 7.17.0 - /usr/local/bin/npm
  npmPackages:
    nuxt: ^2.16.2 => 2.16.2

Reproduction
https://stackblitz.com/edit/github-a2pdyd?file=modules/addpage/_.vue
Describe the bug
I'm not sure if it is a bug but,  when there is no 'pages' directory in the Nuxt.js application, modules cannot add or extend routes.
Please check reproduction. There is no pages directory and we can see
 WARN  No pages directory found in /home/projects/github-a2pdyd. Using the default built-in page.

But if you add pages directory then page file from module will be applicable.
I have an idea to check if the 'pages' directory doesn't exist and then change the pages directory.
Is there any other option to force enabling it?
Here is my temporary solution
if (!fs.existsSync(this.options.srcDir + '/' + this.options.dir.pages)) {
    this.options.dir.pages = relative(
      this.options.srcDir,
      resolve(__dirname, './pages/')
    );
  }

Additional context
No response
Logs
No response

We are approaching the Nuxt 2 EOL date (June 30, 2024) - see this article for more information. This is advance warning that I'm going to close this issue then, as it's currently marked as a Nuxt 2 related bug.
If it's a critical or security issue, please do comment and let me know, in case it is possible to address it before the EOL date.
If it's a an issue you think is relevant for Nuxt 3, please feel free to open a fresh issue (or just comment here so I can update labels, etc.). 🙏
Thank you for your understanding and for your contribution to Nuxt. ❤

It's the day, at last. Nuxt 2 is now marked end-of-life.
My apologies we never got round to resolving this bug in the 2.x branch, but we do have to draw a line somewhere. Again, if you think this is still relevant for Nuxt going forward, please feel free to open a fresh issue (or just comment here so I can update labels, etc.). 🙏


How Can we open OOTB SharePointOnline List Display, Edit, and New forms in the spfx web part programmatically from the spfx web part

Category

[X] Question

Question

I would like to know How can I open OOTB SharePoint List Display, Edit, and New forms in the spfx web part programmatically from the spfx web part in SharePoint Online?

I have three button  Edit, New, Display in My Custom web part, and want to open the Right Side Panel with the appropriate form.
Thanks in advance for Any suggestion or way.

Hi, AFAIK, there isn't a way to do this. You will have to use the Fluent UI Panel to launch yourself as this isn't something that is callable.  Second, it may be worth looking into the React-List-Form web part from the SP-Dev-fx-webparts repository and see if you can modify it to fit within your solution. https://github.com/pnp/sp-dev-fx-webparts/tree/master/samples/react-list-form

https://sharepoint.uservoice.com/forums/329220-sharepoint-dev-platform/suggestions/37169995-allow-us-to-develop-custom-modern-forms-with-custo
We used JSLinks to set redirection to a custom page with SPFx customization, but because of MC222880 it's no longer possible. MS Support advised us to use PowerApps, it's not possible in our case. So, I'm curious too what else is using by the community.

Any news on this request?


Add uart config logging

Dumps actual UART configuration into debug log
(feel free to close this one)
Logging the actual configuration of port turned to be super helpful, when working with different then default UART config. I got cought by the fact, that modbux and Circuits.UART defaults are different, as I've initially tried to pass only settings different to modbux defaults. What also added to confusion was the fact, that initially, I though that defaults are per attribute and not for uart_opts as a whole.
Sorry for the other included commit. It followed #9
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Thanks @bobek, I think is a good idea to log the uart configuration parameters for troubleshooting.


Issue with _docs/tasks/traffic-management/ingress.md

Following the wiki, it looks like I have to create two ingresses to be able to allow http and https access.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: secure-ingress
  annotations:
    kubernetes.io/ingress.class: istio
spec:
  tls:
    - secretName: istio-ingress-certs # currently ignored
  rules:
  - http:
      paths:
      - path: /status/.*
        backend:
          serviceName: httpbin
          servicePort: 8000
      - path: /delay/.*
        backend:
          serviceName: httpbin
          servicePort: 8000

This ensures access to httpbin with tls termination at istio ingress.
kubectl describe ingress secure-ingress does show up both ports 80 and 443.
But, I have to create the following ingress as well so as to be able to access using http.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: simple-ingress
  annotations:
    kubernetes.io/ingress.class: istio
spec:
  rules:
  - http:
      paths:
      - path: /status/.*
        backend:
          serviceName: httpbin
          servicePort: 8000
      - path: /delay/.*
        backend:
          serviceName: httpbin
          servicePort: 8000

When I tested using two ingresses as above using nginx-ingress, the access for the service get disrupted as both the ingress have the same rules for the service (i.e. http). I just need to write one ingress with tls and that took care of both http and https.
Is this the behavior with istio ingress or am I missing something? The documentation doesn't call out if secure-ingress is sufficient for both http and https.

Another effect of this behavior is that istio does not auto upgrade http requests to https. If client requests for http connection to a service that has an ingress with tls, he will get a 404.


fix columns order in returning list

Changelog category

Bugfix
Not for changelog (changelog entry is not required)

#6699


Wrapper or type to avoid inlinng

Functions that take other functions as arguments are an essential part of many numeric libraries, including those dealing with minimizers, solvers and integrators among others. While recent numba version have seen an improvement in relation to this (mainly due to #5579 and #6373) the user experience is still inconsistent.
Consider the following example
# solver library code
def solver(rhs, starting_value):
    # Some cool code here

# user code
def my_problem(x):
    # some code here

solution = solver(my_problem, 0)

Depending on the content of my_problem and solver sometimes the cache will work and sometimes will not (Check this out for a great example of this #6772 )
In my opinion, the problem is that we are not telling numba that rhs is a function pointer and therefore should never be inlined. In other words, we want to be able to compile the solver library ahead of time and independent of what the user will provide.
The current inability to do this makes it very difficult to provide numba libraries that deal with minimizers, solvers and intergrators as there is always a very large compilation overhead that cannot be cached. We are seeing this, for example, in https://github.com/hgrecco/numbakit-ode The integration itself beats SciPy code by a factor of 10 but warming up takes a very long time that makes it unusable for small integrations.
My proposed solution is one of the following:

Treat arguments with FunctionType as function pointers.
Add a boolean flag to FunctionType to indicate if it should be treated like a pointers.
Create FunctionPointerType to indicate that an argument should never be inlined.

As a very frequent user of the argumentless njit I understand that my solutionm might not be allways welcome. Library maintainers will need to explicitely add a signature and library users will need to adhere to it. But in any case, this will still better than coding in Cython or using C, C++ or fortran where we also need to use a signature. In addition, libraries like https://github.com/hgrecco/numbakit-anjit/ could be use to help with adding a signature.
See also:

https://numba.discourse.group/t/using-the-cache/666

@hgrecco Thanks for raising this issue, this is definitely a use case that needs a good solution because it's so common. Will mark this for discussion at tomorrow's development meeting (they are public, everyone is welcome to attend). I think one of 2. or 3. from above could work, will see if there are any more views tomorrow. Thanks again!

Thanks @stuartarchibald Sadly I have a meeting tomorrow at that exact time but I am available for thinking, testing or reviewing, or if can join other meeting.

Thanks for joining the discussion yesterday @hgrecco. I've put together a demonstration of various use cases, hopefully this can aiding in guiding either what is missing or help in finding a solution from what is currently available. CC @CalebBell this may also be of interest to you.
from numba import njit, types
import numpy as np

# ------------------------------------------------------------------------------
# Case 1.
# Optimiser takes objective function, everything is a dispatcher
print('Case 1'.center(80, '-'))

@njit
def optimiser1(objective_func, *args):
    ... # work
    return objective_func(*args)

@njit
def objfunc1(a, b):
    return a / b

print("answer:", optimiser1(objfunc1, 6, 3))
print("optimiser signatures", optimiser1.signatures)
print("objective function signatures", objfunc1.signatures)
print("objective function type", type(objfunc1))

# ------------------------------------------------------------------------------
# Case 2.
# Optimiser takes objective function, everything is a dispatcher but with
# caching. This is fine _but_ look at the machine code, objfunc2 is inlined
print('Case 2'.center(80, '-'))

@njit(cache=True)
def optimiser2(objective_func, *args):
    ... # work
    return objective_func(*args)

@njit(cache=True)
def objfunc2(a, b):
    return a / b

print("answer:", optimiser2(objfunc2, 6, 3))
print("optimiser signatures", optimiser2.signatures)
print("objective function signatures", objfunc2.signatures)
print("objective function type", type(objfunc2))
# Note this is a cache miss every time, the CPU dispatchers change each run
print("optimiser cache stats", optimiser2.stats)
# Also note that in the machine code the objective function has been inlined
# into the optimiser
asm = optimiser2.inspect_asm(optimiser2.signatures[0]).splitlines()
for ix, x in enumerate(asm):
    if 'ZN8__main__' in x and ':' in x:
        break
print("optimiser assembly", '\n'.join(asm[ix:ix + 10]))

# ------------------------------------------------------------------------------
# Case 3.
# Optimiser takes objective function, objective function is declared with a
# signature.
print('Case 3'.center(80, '-'))

@njit
def optimiser3(objective_func, *args):
    ... # work
    return objective_func(*args)

@njit(types.int64(types.int64, types.int64))
def objfunc3(a, b):
    return a / b

print("answer:", optimiser3(objfunc3, 6, 3))
print("optimiser signatures", optimiser3.signatures)
print("objective function signatures", objfunc3.signatures)
print("objective function type", type(objfunc3))

# ------------------------------------------------------------------------------
# Case 4.
# As case 3 but now with caching. Optimiser takes objective function, objective
# function is declared with a signature.
print('Case 4'.center(80, '-'))

@njit(cache=True)
def optimiser4(objective_func, *args):
    ... # work
    return objective_func(*args)

@njit(types.int64(types.int64, types.int64), cache=True)
def objfunc4(a, b):
    return a / b

print("answer:", optimiser4(objfunc4, 6, 3))
print("optimiser signatures", optimiser4.signatures)
print("objective function signatures", objfunc4.signatures)
print("objective function type", type(objfunc4))
# Cache miss every time
print("optimiser cache stats", optimiser4.stats)
# Also note that in the machine code the objective function has been inlined
# into the optimiser
asm = optimiser4.inspect_asm(optimiser4.signatures[0]).splitlines()
for ix, x in enumerate(asm):
    if 'ZN8__main__' in x and ':' in x:
        break
print("optimiser assembly", '\n'.join(asm[ix:ix + 10]))

# ------------------------------------------------------------------------------
# Case 5.
# Optimiser takes objective function and specifies signature for what is
# accepted, objective function left open for recompilation.
print('Case 5'.center(80, '-'))

fnty = types.FunctionType(types.int64(types.int64, types.int64))
@njit(types.int64(fnty, types.UniTuple(types.int64, 2)))
def optimiser5(objective_func, *args):
    ... # work
    return objective_func(*args)

@njit
def objfunc5(a, b):
    return a / b

print("answer:", optimiser5(objfunc5, 6, 3))
print("optimiser signatures", optimiser5.signatures)
print("objective function signatures", objfunc5.signatures)
print("objective function type", type(objfunc5))
# Also note that in the machine code the objective function has NOT been inlined
# into the optimiser
asm = optimiser5.inspect_asm(optimiser5.signatures[0]).splitlines()
for ix, x in enumerate(asm):
    if 'ZN8__main__' in x and ':' in x:
        break
print("optimiser assembly", '\n'.join(asm[ix:ix + 30]))

# ------------------------------------------------------------------------------
# Case 6.
# As case 5 but now with caching. Optimiser takes objective function and
# specifies signature for what is accepted, objective function left open for
# recompilation.
print('Case 6'.center(80, '-'))

fnty = types.FunctionType(types.int64(types.int64, types.int64))
@njit(types.int64(fnty, types.UniTuple(types.int64, 2)), cache=True)
def optimiser6(objective_func, *args):
    ... # work
    return objective_func(*args)

@njit(cache=True)
def objfunc6(a, b):
    return a / b

print("answer:", optimiser6(objfunc5, 6, 3))
print("optimiser signatures", optimiser6.signatures)
print("objective function signatures", objfunc6.signatures)
print("objective function type", type(objfunc6))
# Cache will be hit on subsequent runs
print("optimiser cache stats", optimiser6.stats)

# ------------------------------------------------------------------------------
# Case 7.
# Demonstration of #6772 in light of the above use cases.
print('Case 7'.center(80, '-'))

# The issue here is that `call_solver7` is referring to `method7` and
# `objective_func7` which are globals and so to cache they must be inlinable by
# llvm else they will appear as a function pointer (an address) in the machine
# code that changes across program instances so cannot be cached.

# This works fine, everything is LLVM level inlined.
@njit(cache=True)
def objective_func7(x):
    return x * 10

@njit(cache=True)
def method7(fnptr, *args):
    return fnptr(*args)

@njit(cache=True)
def call_solver7(x):
    return method7(objective_func7, x)

print(call_solver7(3))

# This does not work, the following won't cache as the method7_cannot_cache is
# too large to LLVM level inline
@njit(cache=True)
def objective_func7(x):

    return x * 10

@njit(cache=True)
def method7_cannot_cache(fnptr, *args):
    acc = 0
    for i in range(args[0]):
        acc += i
    acc /= 1.2345
    for i in range(args[0], -1, -1):
        acc += i
    acc /= 1.2345
    for i in range(args[0]):
        acc += i
    acc /= 1.2345
    for i in range(args[0], -1, -1):
        acc += i
    acc /= 1.2345
    for i in range(args[0]):
        acc += i
    acc /= 1.2345
    for i in range(args[0], -1, -1):
        acc += i
    acc /= 1.2345
    # NOTE: This is a "+7" too much and prevents inlining so breaks the cache
    acc /= 1.2345 * acc + 7
    return fnptr(*args) + acc

@njit(cache=True)
def call_solver7_cannot_cache(x):
    return method7_cannot_cache(objective_func7, x)

print(call_solver7_cannot_cache(3))

This gives me:
-------------------------------------Case 1-------------------------------------
answer: 2.0
optimiser signatures [(type(CPUDispatcher(<function objfunc1 at 0x7f025cdc7670>)), UniTuple(int64 x 2))]
objective function signatures [(int64, int64)]
objective function type <class 'numba.core.registry.CPUDispatcher'>
-------------------------------------Case 2-------------------------------------
answer: 2.0
optimiser signatures [(type(CPUDispatcher(<function objfunc2 at 0x7f025b630280>)), UniTuple(int64 x 2))]
objective function signatures [(int64, int64)]
objective function type <class 'numba.core.registry.CPUDispatcher'>
optimiser cache stats _CompileStats(cache_path='__pycache__', cache_hits=Counter(), cache_misses=Counter({(type(CPUDispatcher(<function objfunc2 at 0x7f025b630280>)), UniTuple(int64 x 2)): 1}))
optimiser assembly _ZN8__main__14optimiser2$243E76type$28CPUDispatcher$28$3cfunction$20objfunc2$20at$200x7f025b630280$3e$29$298UniTupleIxLi2EE:
        testq   %r8, %r8
        je      .LBB0_1
        cvtsi2sd        %r8, %xmm0
        cvtsi2sd        %rcx, %xmm1
        divsd   %xmm0, %xmm1
        movsd   %xmm1, (%rdi)
        xorl    %eax, %eax
        retq
.LBB0_1:
-------------------------------------Case 3-------------------------------------
answer: 2
optimiser signatures [(type(CPUDispatcher(<function objfunc3 at 0x7f025b4c8d30>)), UniTuple(int64 x 2))]
objective function signatures [(int64, int64)]
objective function type <class 'numba.core.registry.CPUDispatcher'>
-------------------------------------Case 4-------------------------------------
answer: 2
optimiser signatures [(type(CPUDispatcher(<function objfunc4 at 0x7f025b5225e0>)), UniTuple(int64 x 2))]
objective function signatures [(int64, int64)]
objective function type <class 'numba.core.registry.CPUDispatcher'>
optimiser cache stats _CompileStats(cache_path='__pycache__', cache_hits=Counter(), cache_misses=Counter({(type(CPUDispatcher(<function objfunc4 at 0x7f025b5225e0>)), UniTuple(int64 x 2)): 1}))
optimiser assembly _ZN8__main__15optimiser4$2411E76type$28CPUDispatcher$28$3cfunction$20objfunc4$20at$200x7f025b5225e0$3e$29$298UniTupleIxLi2EE:
        testq   %r8, %r8
        je      .LBB0_1
        cvtsi2sd        %r8, %xmm0
        cvtsi2sd        %rcx, %xmm1
        divsd   %xmm0, %xmm1
        cvttsd2si       %xmm1, %rax
        movq    %rax, (%rdi)
        xorl    %eax, %eax
        retq
-------------------------------------Case 5-------------------------------------
answer: 0
optimiser signatures [(FunctionType[int64(int64, int64)], UniTuple(int64 x 2))]
objective function signatures [(int64, int64)]
objective function type <class 'numba.core.registry.CPUDispatcher'>
optimiser assembly _ZN8__main__15optimiser5$2414E45FunctionType$5bint64$28int64$2c$20int64$29$5d8UniTupleIxLi2EE:
        .cfi_startproc
        pushq   %rbp
        .cfi_def_cfa_offset 16
        pushq   %r15
        .cfi_def_cfa_offset 24
        pushq   %r14
        .cfi_def_cfa_offset 32
        pushq   %r13
        .cfi_def_cfa_offset 40
        pushq   %r12
        .cfi_def_cfa_offset 48
        pushq   %rbx
        .cfi_def_cfa_offset 56
        subq    $24, %rsp
        .cfi_def_cfa_offset 80
        .cfi_offset %rbx, -56
        .cfi_offset %r12, -48
        .cfi_offset %r13, -40
        .cfi_offset %r14, -32
        .cfi_offset %r15, -24
        .cfi_offset %rbp, -16
        movq    %r9, %r14
        movq    %r8, %rbx
        movq    %rdx, %rbp
        movq    %rdi, %r15
        movl    $0, 12(%rsp)
        testq   %rdx, %rdx
        je      .LBB0_1
.LBB0_8:
-------------------------------------Case 6-------------------------------------
answer: 0
optimiser signatures [(FunctionType[int64(int64, int64)], UniTuple(int64 x 2))]
objective function signatures []
objective function type <class 'numba.core.registry.CPUDispatcher'>
optimiser cache stats _CompileStats(cache_path='__pycache__', cache_hits=Counter(), cache_misses=Counter({(FunctionType[int64(int64, int64)], UniTuple(int64 x 2)) -> int64: 1}))
-------------------------------------Case 7-------------------------------------
30
fnptr_cases.py:212: NumbaWarning: Cannot cache compiled function "call_solver7_cannot_cache" as it uses dynamic globals (such as ctypes pointers and large global arrays)
  @njit(cache=True)
30.579463904781125

Quick summary:

The use of signatures which specify a FunctionType on an optimiser function I think gets what you want with respect to caching.
Referring to functions in globals is what runs into problems with respect to caching and the noted reliance on the LLVM inliner.
The warning in relation to uncacheable functions due to pointers etc probably needs to be more specific.

Hope this helps?

Hi Stuart!
Thank you for pulling together these examples. They help to understand what is going on greatly. Using FunctionType is no problem and I have the information to use it everywhere.
I am struggling still with the "referring to functions in globals" bit. Is there any way around that being an issue? Otherwise, we are still never able to cache the algorithm, objective function, and calling function together. The following example shows the issue with the FunctionType information added.
from numba import njit, types
import numba
from math import sqrt

objf_type = types.FunctionType(types.float64(types.float64, types.float64))

@njit(types.float64(objf_type, types.float64, types.UniTuple(types.float64, 1)), cache=True)
def secant(func, x0, args=()):
    p0 = x0
    p1 = x0*0.9999

    q0 = func(p0, *args)
    q1 = func(p1, *args)

    for i in range(100):
        # Calculate new point
        p = p1 - q1*(p1 - p0)/(q1 - q0)

        # Check the exit conditions
        if (abs(p0 - p1) <= abs(1e-9*p0)):
            return p
        # Swap the points around
        p0 = p1
        q0 = q1
        p1 = p
        q1 = func(p1, *args)
    return 0.0

@njit(types.float64(types.float64, types.float64), cache=True)
def to_solve(x, y):
    sln = 0.0
    sln += 10.001*x*x - 2.2*x - y
    # Change x*y*x to x*y to make this function cache
    sln -= (x*y*x - sqrt(x))*1e-40/x**2.5/sqrt(x**2.5)
    return sln

@njit(types.float64(types.float64), cache=True)
def my_solve(y):
    return secant(to_solve, 5.0, args=(y,))

print((my_solve(10.0)))

Sincerely,
Caleb

@CalebBell no problem, glad they help.

I am struggling still with the "referring to functions in globals" bit. I

There's no easy way around the "functions in globals" bit (IIRC there is technically a way because numba does it internally, but it's challenging). When code like this is compiled:
@njit
def bar():
  pass

@njit
def foo():
    bar()

foo()

The function foo() "sees" bar as a global and the machine code for foo refers to bar by address, that address cannot be cached as it will change next time the program is run and so caching is disabled for function foo.
What I'd anticipate numerical solvers to consume is the objective function itself and some parameters, it looks like in your case you are wanting to do something partial application-like and create specialisations of a given solver, given objective function and also some parameters?
I think that this might all boil down to a feature request for caching to support address relocation.

I think that this might all boil down to a feature request for caching to support address relocation.

This is also likely going to manifest as a dependency chain problem too with regards to when a cache has been invalidated.

Thanks @stuartarchibald for the use cases. It is very usefull to see all this together
I think where you wrote optimiser6(objfunc5, 6, 3) you ment optimiser6(objfunc6, 6, 3)
But apart from that, I wonder if there is any (potential) difference in doing this (let's call it 6b)
objfnty = types.int64(types.int64, types.int64)
fnty = types.FunctionType(objfnty)
@njit(types.int64(fnty, types.UniTuple(types.int64, 2)), cache=True)
def optimiser6b(objective_func, *args):
    ... # work
    return objective_func(*args)

@njit(objfnty, cache=True)
def objfunc6b(a, b):
    return a / b

i.e. make explicit the signature for objfunc6b. I was not able to spot any difference when running, but I was curious.
Regarding globals, would it be possible to compile the module not function? Because then the address of bar would be fixed for foo . In other words, as long as I can compile the optimiser6b  and cache it I would be satisfied. For that any global function used by  optimiser6b  needs to be accesible as well.
@CalebBell What do you mean by caching together? To me is easier to think as library code and user code. secant is library code and should be compiled it and cached by the library (at the library pycache so it can be reused) and it would be nice that it can be done through some hook at installing the package. to_solve is user code. I think compiling my_solveis so important at ths stage.

@stuartarchibald one more thing.
I think what I wonder is :
@njit(types.float64(types.float64[::1]))
def mysum(x):
    return np.sum(x + 1.)

Will this function be cached? It is using a global (np.sum) so I guess the answer is no. Am I correct?

Spent a bunch of time looking at machine code and code gen and I think I've found a bug. Numba is correctly refusing to cache case 7 above based on what is generated, it's just that I think what is generated need not contain a dynamic address.

thanks for all the examples @stuartarchibald . It's very useful to see all the combinations. I am surprised at the difference in caching behaviour between inlineable and non inlineable functions. you wrote

This is also likely going to manifest as a dependency chain problem too with regards to when a cache has been invalidated.

but isn't this also a problem with inlineable functions? The cache should be invalidated when the inlined function changes, so how is that dependency handled?

Hi!
I'm a little late for the discussion, but I'd like to support efforts to allow functions as arguments to numba compiled functions. Unfortunately, not being able to cache such functions makes such implementations slower for me as well. I have three libraries all relying heavily on Numba:
https://github.com/yoelcortes/flexsolve
https://github.com/BioSTEAMDevelopmentGroup/thermosteam
https://github.com/BioSTEAMDevelopmentGroup/biosteam
I'd like to show two very similar examples, one that doesn't raise a warning and one that does. I cannot figure out why does the second raise a warning but the first does not:
(Add this module to your python path for caching)
numba_warning_test.txt
Code:
# Test 1:
from numba_warning_test import bisection, false_position, njit
f = njit(lambda x, y: x**3 - 40 + y*x)
g = njit(lambda y, z: y + bisection(f, -100, 100, args=(y,)))
h = njit(lambda z: z - bisection(g, -10, 10, args=(z,)))
x = bisection(h, -5, 5)
# Test 2:
f = njit(lambda x, y: x**3 - 40 + y*x)
g = njit(lambda y, z: y + false_position(f, -100, 100, args=(y,)))
h = njit(lambda z: z - false_position(g, -10, 10, args=(z,)))
x = false_position(h, -5, 5)

Output:
<ipython-input-2-9453669cd16a>:3: NumbaWarning: Cannot cache compiled function "false_position" as it uses dynamic globals (such as ctypes pointers and large global arrays)
  h = njit(lambda z: z - false_position(g, -10, 10, args=(z,)))
<ipython-input-2-9453669cd16a>:3: NumbaWarning: Cannot cache compiled function "false_position" as it uses dynamic globals (such as ctypes pointers and large global arrays)
  h = njit(lambda z: z - false_position(g, -10, 10, args=(z,)))

@stuartarchibald, not sure if this is helpful, but it is related to case 7.
I'm using Python 3.8 on a Windows with numba 0.53.1.
Hope this helps,
Thanks!

I've removed "needtriage" to prevent this discussion being marked stale again.

Thanks @gmarkall. This is probably actually a result of a "bug", so am going to label it as such. The bug is that there's dynamic addresses being needlessly generated.

This seems to fix up most of the issues, but it lacks elegance. https://github.com/stuartarchibald/numba/commit/d8b4e1f9a527346f0839a11cae01af8070667217
The issue is essentially that the experimental first-class function type feature adds a specific lowering for a constant dispatcher here:
https://github.com/numba/numba/blob/e050fd691abcde71ac84dd8f748d158332ff060c/numba/experimental/function_type.py#L64-L67
as a result this very specific lowering matches all constant dispatchers, regardless of whether there's anything to do with first-class function types going on. This lowering emits a dynamic address and it is this dynamic address that is being spotted by the cache implementation which causes it to (correctly) abort caching.

Does it make sense conceptually for a constant dispatcher lowering to have changed after the first-class function implementation?
A constant distpatcher is a constant dispatcher and cannot be another dispatcher, like a constant string is a literal string and not a variable of string type whose value will be determined later.
I would have thought that only input arguments of dispatcher type would have been affected by the first-class function change, because in those cases we say that this particular argument is broader than one specific dispatcher and could by any other dispatcher that has the same signature.
I don't know anything about lowering of functions, so I could be completely off here. I am only thinking conceptually, and there might be implementation details that I'm ignoring.

Hi Stuart,
This looks awesome to me! Your code changes are clean and easy to read. In my experience every new feature always adds some places that don't tie in very elegantly. Fixing some cases and adding tests for them can lead to opportunities for doing bits of rearchitechting safely later :)

@stuartarchibald I'd like to work on this issue but I don't fully understand your fix in https://github.com/stuartarchibald/numba/commit/d8b4e1f9a527346f0839a11cae01af8070667217
Why must a dispatcher become non cacheable after unification? The comment here https://github.com/stuartarchibald/numba/commit/d8b4e1f9a527346f0839a11cae01af8070667217#diff-8c961fdf472869ea0052bda667a6f4adc50d0637fb0cec47861964d81fab3de8R1241 suggests that it is not a conceptual problem but only done to play nice with the code in https://github.com/stuartarchibald/numba/commit/d8b4e1f9a527346f0839a11cae01af8070667217#diff-399e8808199a3d1d28292c1501afcee207b1586916ad34c0bda3774616657f17R66-R70
what's the purpose of the dynamic address? If a dispatcher is ever cacheable (thus allowing context.get_dummy_value() instead of add_dynamic_addr, then I don't see why the first-class function should change that. Conceptually I would argue that a constant dispatcher is never a first-class function type, and therefore the unification should not affect the cacheability.
I don't fully understand how the lowering of functions work, but maybe with a bit more knownledge I could work on this issue.


sidenotes appearing on the left

Since the resolution of issue #1 a new bug has appeared. Now, sometimes the sidenotes appear on the left, instead of the right. Fix should be a simple matter of adding the float-right classes to those elements.

fixed by d2ece8d
(typo in commit message there. #2 is not related to that commit.)


Alloc 2tier

Add support for "global.ddr" memory scope for Hexagon.
(Much better commit message needed.)

@Lunderberg @csullivan @kparzysz-quic : Heads up, this should be ready for review once CI passes.

@csullivan @Lunderberg @adstraw @kparzysz-quic: FYI if anyone wants to review

This might need fixing:
tests/python/contrib/test_hexagon/test_2d_physical_buffers.py::TestElementWise::test_execute[nchw-8h8w32c-1d-nchw-8h8w32c-1d-int8-2-32-8x8-nhwc-global.vtcm-hexagon] [09:37:00] /home/cconvey/r/cconvey/tvm/alloc-2tier/src/target/target_info.cc:45: Warning: MemoryInfo for scope = global.vtcm is undefined
[09:37:00] /home/cconvey/r/cconvey/tvm/alloc-2tier/src/target/target_info.cc:45: Warning: MemoryInfo for scope = global.vtcm is undefined

This might need fixing:
tests/python/contrib/test_hexagon/test_2d_physical_buffers.py::TestElementWise::test_execute[nchw-8h8w32c-1d-nchw-8h8w32c-1d-int8-2-32-8x8-nhwc-global.vtcm-hexagon] [09:37:00] /home/cconvey/r/cconvey/tvm/alloc-2tier/src/target/target_info.cc:45: Warning: MemoryInfo for scope = global.vtcm is undefined
[09:37:00] /home/cconvey/r/cconvey/tvm/alloc-2tier/src/target/target_info.cc:45: Warning: MemoryInfo for scope = global.vtcm is undefined

I discussed this via DM with @csullivan.  His impression is that having supporting MemoryInfo for a given memory-scope can be useful, but isn't required.  So I'm not planning to deal with this warning in this PR.

@csullivan @jverma-quic : request for review

Thank you @cconvey @csullivan @adstraw @nverke, the PR has been merged.


Don't copy hidden files and folders into temp folder

When Terragrunt downloads Terraform configurations from a source URL into a temp folder, it also copies the files in the current working directory (e.g. terraform.tfvars) into that temp folder. Unfortunately, as part of the copy process, Terragrunt would also copy hidden files and folders, such as .git or .terraform, which would lead to confusing errors. This PR updates Terragrunt to ensure it no longer copies hidden files & folders.

This is a fairly simple change with reasonable tests, so I'm going to merge. Feedback welcome!


Please invite me to the GitHub Community Organization

Name
Cleanninja
Discord Username (if applicable)
No response
Additional Context
Please join me

It's great having you contribute to this project
If you would like to continue contributing to open source and would like to do it with an awesome inclusive community, you should join our Discord Server and our GitHub Organisation - we help and encourage each other to contribute to open source little and often 🤓 . Any questions let us know.

Invitation sent for the GitHub Organisation. Welcome to the community 🎉#
Don't forget after accepting to make it public so it appears on your GitHub profile for everyone else to see, you can do this by finding your name in the GitHub organisation list and change the dropdown to public https://github.com/orgs/jaidevstudio/people
Like this

Tips for practising:
Remember contributing to open source is not just about code, its about collaboration, communication and adding value
I hope that helps

Please check your email or organisation page 🙏


[cluster-test] Change some println -> slog macro

@bors-libra r+

:pushpin: Commit 2fadf18 has been approved by ankushagarwal

:hourglass: Testing commit 2fadf188f909bfe1adaf5a4adc67bce5847e18d3 with merge f8460b9286d5b863e5aebe0266f6a87437f51950...

:sunny: Test successful - checks-circle_commit_workflow
Approved by: ankushagarwal
Pushing f8460b9286d5b863e5aebe0266f6a87437f51950 to master...


Fixed node playwright image

Renamed steps to make sense.

The problem was that playwright tried to access the browser folder outside the myuser which resolved in access denied. The fix is in overriding the playwright env var to store the browsers inside the myuser.


Wrong changes in the change history

Hello colleagues!
We are working on the project https://github.tools.sap/erp4sme/crypto-for-business.
The issues that we have are referring to the change tracking; however the difference to be reflected in the change history is being estimated in the utilities of node_modules/@sap/cds/libx/_runtime/common/utils.
We have an app:
App annotation   app\payment\downstreamPaymentProcessors\annotations.cds
Entity CDS           db\payment\paymentRails\DownstreamPaymentProcessors.cds
UI service CDS     srv\payment\paymentRails\ui-service-downstreamPaymentProcessors.cds
UI service provider srv\payment\paymentRails\ui-service-downstreamPaymentProcessors.ts
Entity DownstreamPaymentProcessors has compositions to two views:
entity DownstreamPaymentProcessors : cuid, managed {
@changelog
@title    : '{i18n>Name}'
name                   : String;
@title    : '{i18n>Integration}'
@changelog: [Integration.name]
Integration            : Association to one Integrations;

@title    : '{i18n>ActivationStatus}'
@changelog: [ActivationStatus.name]
ActivationStatus       : Association to one codelists.PaymentProcessorActivationStatusCodes;
callbackSupported      : Boolean;
NetworkMappings        : Composition of many DownstreamPaymentProcessorNetworkMappings
                             on NetworkMappings.contextObjectId = $self.ID;
CryptocurrencyMappings : Composition of many DownstreamPaymentProcessorCryptocurrencyMappings
                             on CryptocurrencyMappings.contextObjectId = $self.ID;

}
entity DownstreamPaymentProcessorNetworkMappings        as
select from GeneralMappings {
*,
Network : Association to one Networks on Network.ID = sourceObjectId,
}
where
contextEntityType.code = 'PAYMENT_PROCESSOR'
and sourceEntityType.code  = 'NETWORK';
entity DownstreamPaymentProcessorCryptocurrencyMappings as
select from GeneralMappings {
*,
Cryptocurrency : Association to one Cryptocurrencies on Cryptocurrency.ID = sourceObjectId,
}
where
contextEntityType.code = 'PAYMENT_PROCESSOR'
and sourceEntityType.code  = 'CRYPTOCURRENCY';
These Mappings entities are also annotated for change tracking.
Although UI service entity is declared as draft enabled, all DB operations during SAVE event are fulfilled explicitly via wrapper service (srv\payment\wrapper\wrapper-service.cds).

The issue with the change history is:

Whenever a UI change is done on mappings (for example a new mapping is added) then the mappings which have existed so far are shown as if they were deleted (although they are not):

I saw:
In<EMAIL_ADDRESS>the function _addToBeDeletedEntriesToResult set _op: 'delete' on the mappings.

I tested also an alternative way and removed calling DB operations during SAVE – i.e. let drafts to be activated per standard draft handling (I did it in the branch mappings-change-history).
Then I saw that unlike the previous solution<EMAIL_ADDRESS>did not get any old value to compare with the new one. Then mappings were marked with _op: 'create' – as if they were new.

Additional issue:
For the entity DownstreamPaymentProcessor some fields are shown as updated in every change log, although they were not changed:

Could you please help? We think that the issues began after upgrading the CDS version to CDS8 (we have 8.4.2 now).
Kind regards,
Elena Gurevitch

@Sv7enNowitzki is this related to the DATE-Issues with the DB driver that we have?

@Sv7enNowitzki is this related to the DATE-Issues with the DB driver that we have?

Regarding the third issue mentioned in the ticket, I suspect it might be the case, as it seems that C4B has started using the new driver @cap-js/hana after upgrading to cds8
Best Regards,
Wenjun

We are watching different behavior of the change history also in other apps. I have an impression, evrything due to upgrade.
For example in two apps no any "create" changes are shown at all, only "update" - the entities were created via wrapper service and updated also via wrapper service, but within job execution.
Earlier (before upgrade, in the previous release) we had issues with missing change entries for job execution. Then they were solved.
Now it is somehow vice versa.
Kind regards,
Elena

Hello colleagues,
I found an additional issue. In some apps "create" changes are not shown.
I found even the reason. For example, for the "Wallets" app:

Change history is being written on "before CREATE" event. Wallet is created per action; INSERT is executed with ID = null).
So at the time when change history should be written there is no any entity ID - that is the reason.
If we create UUID for example in UI service and send it in the request then it will work.

However it would be rather a workaroud because an entity can be created without ID and ID is being generated during INSERT.
The questions are:

Why changelog is being inserted on "before" event?
How is it possible that the same used to work before upgrading CDS and Hana DB?

Kind regards,
Elena

P.S. As I also supposed, before upgrade ID was filled on "before CREATE" event.
Could you please test the same?

The questions are:

Why changelog is being inserted on "before" event?
How is it possible that the same used to work before upgrading CDS and Hana DB?

Kind regards, Elena

Hello Elena,
I can only guess that it's to do with the new DB drivers that were introduced with CDS 8.
I will need to do a much deeper digging, and a fix won't be straightforward, so the best thing we could do right now would be to document the limitation and workaround in the readme for the time being.
Best,
Nick

Hi Nick,
The only workaround I found was to generate an ID and fill it before inserting an entry.
Kind regards,
Elena


Initial Documentation

Description

Related Issue

Tasks

[ ] I have signed the CLA.
[ ] I have written tests and verified that they fail without my change.

The generated documentation looks really good.  Thanks for doing all this.  I think you should merge this PR.
But then I have some questions/issues we should look at:

Will we be able to host these html docs on our github pages?  In my experience, those pages do not allow you to import JavaScript files from your build -- they come across with the content type text/plain
Some variable references didn't process correctly? e.g. $value in: Returns the absolute value of the provided argument $value
Some references to a null return value are linked to null()
The JS method jsonFormula() is documented along with expression functions
Will we be able to fold in the (modified) JMESPath documentation?

I did not receive the notification about these comments and hence forgot to reply to them.

Will we be able to host these html docs on our github pages ?

I think so. The tutorial page also refers to JavaScript. But I will check on my local fork and then merge this PR.

Some variable references didn't process correctly?
Some references to a null return value are linked to null()
The JS method jsonFormula() is documented along with expression functions

Will fix that

Will we be able to fold in the (modified) JMESPath documentation?

Not sure understood this part, but happy to discuss it tonight. Will schedule a meeting.
I will merge this PR by today.

Link to hosted documentation https://vdua.github.io/json-formula/dist/jsdocs/index.html


fix(package): update react-transition-group to version 2.0.2

Closes #11

Coverage remained the same at 98.193% when pulling 7558fc998382cf4d79d1060d724665d958d8a277 on greenkeeper/react-transition-group-2.0.2 into d592aea4f5bf1dcbbc8b9875cf8fee6be438f882 on master.

Coverage remained the same at 98.193% when pulling 7558fc998382cf4d79d1060d724665d958d8a277 on greenkeeper/react-transition-group-2.0.2 into d592aea4f5bf1dcbbc8b9875cf8fee6be438f882 on master.

Coverage remained the same at 98.438% when pulling f3fd7c226a499ca5defb016ee98181f2deb9e561 on greenkeeper/react-transition-group-2.0.2 into b4be50708d7a41f42b3ee545a69b2fa81af75a30 on master.

do we even need this lib? i might just remove it

Coverage decreased (-0.06%) to 98.378% when pulling 6cebf74cdda2b52cb1c5e7592a4c78412b7be1b8 on greenkeeper/react-transition-group-2.0.2 into b4be50708d7a41f42b3ee545a69b2fa81af75a30 on master.

Coverage remained the same at 98.378% when pulling afe06b156760cea4028e6b93fba6d949cc140e63 on greenkeeper/react-transition-group-2.0.2 into 7677df1d22cfd5fab40227c094cc914ef5bdb23f on master.

Coverage dip unrelated to package removal. Merging.


Developing now playing template for android auto

We want to use this library for both ios and android. Now template is working with ios  but not supported in android.
We would like to take this library and modify it so that it supports now playing screen in android as well.
But we are not sure how to proceed with it.
Can anyone please guide us on how to develop feature or if there is some work around would be great.

Which media player are you planning to use?
We use react-native-track-player for audio playback. For iOS it integrates nicely with this package and for Android Auto there's this branch of react-native-track-player which supports lists and a NowPlaying screen for Android.

Thanks for the response @DanielKuhn
We have request to use different player than you mentioned above, we want to use exo player for android and AV player for ios.

have you reviewed google's android auto guidelines? in short you need to expose the mediaSession token used when building exoplayer, and add in appropriate functions while declaring your exoplayer's foreground service as a MediaLibraryService. This is really out of scope for a standalone library that builds generic car templates, but rather the audio library's job; eg. see audio_service from the flutter community as an example.

Which media player are you planning to use? We use react-native-track-player for audio playback. For iOS/CarPlay it integrates nicely with this package and for Android Auto there's this branch of react-native-track-player which supports lists and a NowPlaying screen for Android.

Hi, I am developing a react native app using react-native-track-player, i want my app to work on android-auto. I tried using react-native-carplay, byt NowPlayingTemplate is not woking.
If you have any example application with reactnative, please share the repo, it will help alot.


Expand Functionality

Need to add:

Additional 'aggressive' scan modes.
Create a few custom nmap scripts to supplement these scan modes.

Accomplished item 1 with the most recent v 1.2


Add Gillick cards and other tweaks

Review commit by commit.
Don't display "Not provided" option in Gillick route

Gillick card on confirm page

Redirect to vaccination page with flash

https://github.com/nhsuk/record-childrens-vaccinations/pull/384 fixes the CI build on this.


Add sales-dashboard demo

This change is

Reviewed 7 of 10 files at r1.
Review status: 7 of 10 files reviewed at latest revision, 4 unresolved discussions.

src/common/demo-area.html, line 15 at r1 (raw file):
Quoted 4 lines of code…
>       :host ::slotted(*) {
>         display: block;
>         overflow-x:hidden;
>       }

is this needed?
the hack for highcharts should be on the chart container only
demo seems to work correctly without this, but I might be missing something

src/sales-dashboard/demo-element.html, line 1 at r1 (raw file):

<link rel="import" href="../../bower_components/polymer/polymer-element.html">

is this file needed?

src/sales-dashboard/sales-dashboard.html, line 15 at r1 (raw file):

import not needed

src/sales-dashboard/sales-dashboard.html, line 31 at r1 (raw file):

    /*display: block;*/

padding: 10px;
overflow-x: hidden;

looks like changes here are not needed

Comments from Reviewable

Review status: 7 of 10 files reviewed at latest revision, 4 unresolved discussions.

src/sales-dashboard/demo-element.html, line 1 at r1 (raw file):
Previously, alvarezguille (Guille) wrote…
is this file needed?

Done.

src/sales-dashboard/sales-dashboard.html, line 15 at r1 (raw file):
Previously, alvarezguille (Guille) wrote…

import not needed

Done.

src/sales-dashboard/sales-dashboard.html, line 31 at r1 (raw file):
Previously, alvarezguille (Guille) wrote…

    /*display: block;*/

padding: 10px;
overflow-x: hidden;

looks like changes here are not needed

Right, it's using card class to style these

Comments from Reviewable

Review status: 7 of 10 files reviewed at latest revision, 4 unresolved discussions.

src/common/demo-area.html, line 15 at r1 (raw file):
Previously, alvarezguille (Guille) wrote…

  :host ::slotted(*) {
    display: block;
    overflow-x:hidden;
  }

is this needed?
the hack for highcharts should be on the chart container only
demo seems to work correctly without this, but I might be missing something

Done

Comments from Reviewable

Reviewed 4 of 4 files at r2, 1 of 2 files at r3.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


[v2] {children: true} for updates doesn't work

{children: true} works well for retrieval, but it doesn't work for updates, my guess is it probably doesn't work for deletes either. It works if you use native base meteor collection because it doesn't understand inheritance anyway.

Oh yes it's definitely not working. I forgot to take that use case into account. I will prepare a fix.

Two more considerations:
SomeAstroClass.find({}, {fields: {}, {children: true}) will this work?, since we aren't selecting the type field. I am going to test this out but my suggestion is this should automatically add the type field to the fields object.
SomeAstroClass.find({}) should return all types by default and cast them to the base "SomeAstroClass", this isn't the case right now.
Let me know your thoughts.

Nothing on the bug, but what does {children: true} do? I couldn't find it in the docs

@jhuenges It's an undocumented feature at the moment, read up on #368

I've fixed it and it will be published in 2.1

@jagi can you let me know your thoughts on the 2 other considerations I posted.

Can you tell me exactly which one?

Two more considerations:
SomeAstroClass.find({}, {fields: {}, {children: true}) will this work?, since we aren't selecting the type field. I am going to test this out but my suggestion is this should automatically add the type field to the fields object.
SomeAstroClass.find({}) should return all types by default and cast them to the base "SomeAstroClass", this isn't the case right now, which seems right because we have "children: true" instead maybe we need "cast: true", and "children: true" should be default?
Let me know your thoughts.

Oh yes good idea to add the type field automatically. I will add that feature.
I'm not quite sure about that one. I can agree that it might return all children automatically, so default would be children: true. However, I would still cast all children to their appropriate classes. I don't want to cast it to parent class.

+1 for children: true by default, and another +1 to keep children their appropriate classes intead of their parent class.

The problem i face is that if you need count of all inherited objecta you have to fetch and count on individual Astro Classes, which can get tedious, Maybe if a workaround for this is added.

Animal = Class.create({ /* ... */ });
Dog = Animal.inherit({ /* ... */ });
Cat = Animal.inherit({ /* ... */ });
Animal.find().count(); // Count all
Dog.find().count(); // Count only dogs
Cat.find().count(); // Count only cats

and of course the children flag will be set to true by default

If that works, we are all set. Is this in master?

The only one thing to implement is default value of the children option. Everything else is already on the v2 branch. So I don't think that there is a need for creating another thread for that.

Awesome ;)

@jagi can you please get this on v2 branch so i can clone it for local usage

@talha-asad it is on the v2 branch

I meant the other part of having it default to children: true. I am using the latest clone and it doesn't have it.

Oh maybe I forgot to push it

Ok pushed


gradle-8.0.1 fails to build gatk: it fails to find one plugin in the cache why building gatk offline

Expected Behavior
The FreeBSD port biology/gatk first builds the gradle cache and then builds gatk offline.
The offline build fails:
Welcome to Gradle 8.0.1!

Here are the highlights of this release:
 - Improvements to the Kotlin DSL
 - Fine-grained parallelism from the first build with configuration cache
 - Configurable Gradle user home cache cleanup

For more details see https://docs.gradle.org/8.0.1/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

FAILURE: Build failed with an exception.

* Where:
Build file '/wrkdirs/usr/ports/biology/gatk/work/gatk-<IP_ADDRESS>/build.gradle' line: 15

* What went wrong:
Plugin [id: 'de.undercouch.download', version: '4.1.2'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'de.undercouch.download:de.undercouch.download.gradle.plugin:4.1.2')
  Searched in the following repositories:
    Gradle Central Plugin Repository

The cache build command:

cd /usr/ports/biology/gatk/work/gatk-<IP_ADDRESS> && /usr/bin/env XDG_DATA_HOME=/usr/ports/biology/gatk/work  XDG_CONFIG_HOME=/usr/ports/biology/gatk/work  XDG_CACHE_HOME=/usr/ports/biology/gatk/work/.cache  HOME=/usr/ports/biology/gatk/work PATH=/usr/local/libexec/ccache:/usr/ports/biology/gatk/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin PKG_CONFIG_LIBDIR=/usr/ports/biology/gatk/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe  -fstack-protector-strong -fno-strict-aliasing "  CPP="cpp" CPPFLAGS=""  LDFLAGS=" -fstack-protector-strong " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing  "  MANPREFIX="/usr/local" CCACHE_DIR="/tmp/.ccache" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" gradle  --gradle-user-home /usr/ports/biology/gatk/work/gradle-gatk --project-cache-dir /usr/ports/biology/gatk/work/gradle-gatk   --build-cache localJar

The offline build command that fails:

cd /usr/ports/biology/gatk/work/gatk-<IP_ADDRESS> && /usr/bin/env XDG_DATA_HOME=/usr/ports/biology/gatk/work  XDG_CONFIG_HOME=/usr/ports/biology/gatk/work  XDG_CACHE_HOME=/usr/ports/biology/gatk/work/.cache  HOME=/usr/ports/biology/gatk/work PATH=/usr/local/libexec/ccache:/usr/ports/biology/gatk/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin PKG_CONFIG_LIBDIR=/usr/ports/biology/gatk/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe  -fstack-protector-strong -fno-strict-aliasing "  CPP="cpp" CPPFLAGS=""  LDFLAGS=" -fstack-protector-strong " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing  "  MANPREFIX="/usr/local" CCACHE_DIR="/tmp/.ccache" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" gradle  --gradle-user-home /usr/ports/biology/gatk/work/gradle-gatk --project-cache-dir /usr/ports/biology/gatk/work/gradle-gatk  --offline --build-cache localJar
Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reused, use --status for details

gradle-8.0.1
gatk-<IP_ADDRESS> and gatk-<IP_ADDRESS>
FreeBSD 13.2
Current Behavior
see above
Context (optional)
No response
Steps to Reproduce
see above
Gradle version
8.0.1
Build scan URL (optional)
No response
Your Environment (optional)
No response

Sorry that you're having trouble with Gradle!
We appreciate the effort that went into filing this issue, but we must ask for more information.
As stated in our issue template, a minimal reproducible example is a must for us to be able to track down and fix your problem efficiently. Our available resources are severely limited, and we must be sure we are looking at the exact problem you are facing.
If we have a reproducer, we may be able also to suggest workarounds or ways to avoid the problem.
The ideal way to provide a reproducer is to leverage our reproducer template. You can also use Gradle Project Replicator to reproduce the structure of your project.
This issue will be closed after 7 days unless you provide more information.


Rootless podman service fails to start

Describe the bug
Systemd user service podman.service for rootless podman will show the following errors when started:
Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH
When I add shadow package to override the path of the systemd user service, a new error would occur:
newuidmap: write to uid_map failed: Operation not permitte
should have setuid or have filecaps setuid: exit status 1

Steps To Reproduce
Steps to reproduce the behavior:

Set virtualisation.podman.enable = true in /etc/nixos/configuration
Start podman.service after login.

Expected behavior
The user service podman.service should be started without error.
Notify maintainers

@zowoq @pennae
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.2.5, NixOS, 23.05 (Stoat), 23.05pre462605.5a1dc8acd97`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

545c7a31e5d

There have been a few changes to podman since this commit, please update your nixpkgs.

Hi, Thanks for your reply. I've updated nixpkgs and updated the metadata in this issue. However, podman.service still fails to start and the error message is the same

After removing the shadow package, it now works without problem. Thank you very much!

I get exactly the same issue.
I'm building and deploying to AWS with morph, pinning nixpkgs to commit
abb2ade261c33516716aa21068d8c10c48d03367.  Should I be using a later commit?
[flatland@ec2-lucia:~]$ nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.101, NixOS, 22.11 (Raccoon), 22.11.3139.67f26c1cfc5`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.1`
 - channels(root): `"nixos-22.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`


PP-0: Drupal and platform updates

This will update the Dockerfile, Hel.fi Platform and HDBT base theme, along with a bunch of configurations.
Requires Stonehenge updates:

Go to your stonehenge installation folder

Run git pull && git checkout 3.x
Run make down && make up

Go to your project directory

Run make down && make up

Run updates on local site:

Run make new
After everything is done, check that all modules are up-to-date: https://helsinki-paatokset.docker.so/fi/admin/reports/updates
Click around on the site and check that nothing is broken (we'll probably need to reimplement some site specific configuration)

Näyttää hyvältä


Add deprecation warning for not autoremembering flags

It'd be better if I squashed these commits...

I'd prefer to call it options instead of flags

We may also want to add a setting that users can toggle with bundle config to stop showing this warning.


Comprehensive corruption detection for workflow storage

Here are all the cases that a workflow run(with unique runID) should be considered as "corrupted":

a) a run has both current record and mutableState, workflowState is open, but history is in-completed(having holes/gaps)

b) a run has both current record and mutableState, workflowState is closed, but history is in-completed(having holes/gaps)

c) a run has only current record but no mutableState

d) a run has only mutableState but no current record, workflow state is open
Today our workflow scanner only detecting corrupted for a) but not doing any actions on them

In addition to a) , workflow can also be considered as corrupted when history is completed but timerMap/activityMap/etc are not consistent with history. This could be solved by https://github.com/uber/cadence/issues/2921

Be careful that deleting a workflow probably needs to be implemented through history engine. Otherwise it's not safe or very complicated.


1.18.2 - [Porting Lib] failed to reload modded shaders leads to game crash

Describe the Bug
My friends and I recently updated our fabric server to match currently available version of Create (0.5.1c, build1159) for 1.18.2 from Modrinth. Server is running fabric 0.14.24 and Create is only server-side mod that causes any gameplay changes. Server runs fine however client keeps crashing even though I downgraded FAPI to 0.76.x as per #1207.
Reproduction Steps

Install fabric (any version starting from 0.14.9 and up to current 0.15.3)
Install fabulously optimized
Install Create
Try to launch the game
Game crashes with [Porting Lib] failed to reload modded shaders
...

Expected Result
Never encountered any issues before upgrading first Create from 0.5.0i to 0.5.1c and then upgrading Fabric to 0.14.24
Screenshots and Videos
No response
Crash Report or Log
https://mclo.gs/O3LEcG8
Operating System
Win10, MacOS
Mod Version
0.5.1c
Minecraft Version
1.18.2
Other Mods

Advancement Info
Animatica
AntiGhost
Architectury
BetterMountHUD
BetterPingDisplay
Borderless Mining
Capes
CIT Resewn
Cloth Config v6
Colormatic
C2me
Contiunity
Create
Cull Less Leaves
Custom Entity Models
DCCH
Debugify
Dynamic FPS
Dynamic Fullbright
EMI + addons / REI + addons (it may sometimes crash with either of those)
FAPI 0.76.x
Fabric Language Kotlin
Fabrishot
Farsight
FerriteCore
Indium
Iris
Item model fix
Jade+ addons
Journeymap
Krypton
LambdaBetterGrass + DynLights
Language Reload
LazyDFU
Light Overlay
Lithium
Memory Leak Fix
Midnight Controls
MixinTrace
Mod menu
MoreChatHistory
NoFade
Proxy Server
Reese's sodium options + sodium extra
Smooth Boot
Sodium + Tempfix
Starlight
ToolTip fix
YOSBR
Zoomify

Additional Context
Installing Not Enough Crashes just sends me in a loop of endlessly reloading Mojang screen and keeps spamming my log with [Porting Lib] failure. Just tried to disable every mod except for FAPI and Create: latest

the log shows you have fabric api 0.77

the log shows you have fabric api 0.77
Nope, it is disabled through Prism. The only running now is 0.76

arguing with the log is useless:
- fabric-api 0.77.0 1.18.2

arguing with the log is useless:
- fabric-api 0.77.0 1.18.2

I am very sorry but that's not right. 0.76 is the only other mod besides Create which is enabled.

then your log is the wrong one

Lemme try to reinstall them. Maybe prev version of my client had FAPI from other source since i got it from my friend

Yeah, seems like FAPI 0.76 I got was mixed up or IDK. Reinstalled it manually via modrinth site and not through launcher, so far so good. Sorry that happened

if your issue is resolved please consider closing this issue


Small bug in Silvercrest/lidl air fryer #2298

Sorry I didn't catch this when I confirmed it as working.
The timer has 2 phases, that is both updated in "cooking time" and "time remaining"
1st is the selected cooking time that counts down to zero
2nd a cooling time is added when cooking time reaches zero. This is always 60s
The bug is only a unit mismatch, as 60min is added to time remaining and cooking time instead of 60 seconds.
Don't know if this is even fixable.

The integration does not add anything, it just displays the timers that are available.  If you can point specifically to which entity has the wrong units, it may help me understand what is wrong.

The sensor.xxx_airfryer_cooling_countdown
Is correctly in seconds.
It goes to 60s after the cooking phase reaches 0 and that is correct but to sensor.xx_airfryer_time_remaining and number.xxx_airfryer_cooking_time are in the same process added 60min

In that case, it seems to be the device itself that is sending the incorrect info.


tests: implement test for multiple event processors

General question: I see a lot of copying fixtures to the lower hierarchy conftest.
Could we go for a more generic approach with less redundancy?
I'm thinking of reducing redundancy by creating utility functions for one.
The other question is, wouldn't it be possible to just adapt whats needed in the nested folder (like adapting the config) and other fixtures like agent would be from the upper conftest with the new config? I'm sure pytest would support something like this.


FSDP 0 loss immediately

System Info
- `Accelerate` version: 0.22.0.dev0
- Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Numpy version: 1.24.4
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 1007.76 GB
- GPU type: NVIDIA A100 80GB PCIe
- `Accelerate` default config:
        Not found

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction
training with current accelerate HEAD with FSDP results in 0 loss on step 0
{'loss': 0.0, 'learning_rate': 3.0000000000000004e-08, 'epoch': 0.0}
{'loss': 0.0, 'learning_rate': 1.2000000000000002e-07, 'epoch': 0.0}
{'loss': 0.0, 'learning_rate': 2.7e-07, 'epoch': 0.0}
  0%|                                                                                | 3/6180 [00:11<6:08:58,  3.58s/it]

reinstalling only accelerate to 0.21.0 has the expected non-zero training loss
{'loss': 7.4443, 'learning_rate': 3.0000000000000004e-08, 'epoch': 0.0}
{'loss': 7.263, 'learning_rate': 1.2000000000000002e-07, 'epoch': 0.0}
{'loss': 6.6462, 'learning_rate': 2.7e-07, 'epoch': 0.0}

this likely stems from either https://github.com/huggingface/accelerate/pull/1745 or https://github.com/huggingface/accelerate/pull/1753 which were merged in the past 3 days and I wasn't having this issue on Thursday with accelerate installed from HEAD.
Expected behavior
training loss not to be 0.0 on the first step

here are the settings I'm using in my trainer:
fsdp:
  - full_shard
  - auto_wrap
fsdp_config:
  fsdp_offload_params: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer

Additional debugging isolates this to the use case where https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html is used. The loss behaves as expected it not enabled. But this should be a regression in accelerate since it was working before.

cc @pacman100

Hello, A minimal reproducer would be helpful

@muellerzr who worked on the PR #1749

@winglian we need a reproducer to test this please, as that PR maintains the same defaults as PyTorch (and the old defaults) and you shouldn't have seen an issue.

@winglian we need a reproducer to test this please, as that PR maintains the same defaults as PyTorch (and the old defaults) and you shouldn't have seen an issue.

It was reproduced by me and one other person using fresh installs of latest full release pytorch, transformers, etc.
What is a reproducer?

@teknium1 a reproducer is a minimal chunk of code we can run on our end to recreate the bug, to help us debug and verify we have fixed the issue when making a fix

For the time being I can't reproduce this on my systems, due to an issue with bits-and-bytes 😕 https://github.com/TimDettmers/bitsandbytes/issues/620
So, for now it'll have to be I push some code and we see if it'll run I'm sorry 🙏
But, as a first try, can either @teknium1 or @winglian install accelerate with pip install git+https://github.com/huggingface/accelerate@autocast and see if that solution works?

That branch didn't seem to work either. Just going to add the notes I posted in Twitter about this only being an issue with Llama-2 for posterity. This doesn't seem to be an issue with legacy Llama as simply changing back to that model doesn't seem to have this problem.

@winglian again any chance you could share the code you are using for these for me to be able to test against? Otherwise I'm flying blind at finding the problems which makes it rather difficult.

Otherwise, since we may have to do a back and forth, try building via pip install git+https://github.com/huggingface/accelerate@autocast-fix, and I'll ping in here when to rebuild

@muellerzr it's this trainer https://github.com/openaccess-ai-collective/axolotl on this branch packing-attn-limit-fa2-rebased.  I'm using this config https://gist.github.com/winglian/e803bf5e305d893de3e68b051189f346 . My apologies as this particular "edge case" has a lot of nuance to it so it's hard to boil it down to a simple reproducing script.
I'll give that branch a try shortly and report back.

the autocast-fix branch results in
  File "/workspace/axolotl/scripts/finetune.py", line 361, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1531, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1641, in _inner_training_loop
    self.model = self.accelerator.prepare(self.model)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1271, in prepare
    result = tuple(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1272, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1084, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1371, in prepare_model
    new_forward = autocast_context(model_forward_func)
TypeError: 'nullcontext' object is not callable

@muellerzr Is there a reason DistributedType.FSDP should not be put into this clause https://github.com/huggingface/accelerate/blob/autocast-fix/src/accelerate/utils/modeling.py#L1430-L1435 ?
Adding it there seems to have brought some parity with Flash Attention 2 so far. I haven't gone back and checked the edge case yet.

That error actually helps a ton which is excellent :) I'll push some more fixes tommorow morning for us to try

@winglian let's try again please :) You should be able to just run the same pip install command

@muellerzr new error
  File "/workspace/axolotl/scripts/finetune.py", line 361, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1531, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1641, in _inner_training_loop
    self.model = self.accelerator.prepare(self.model)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1271, in prepare
    result = tuple(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1272, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1084, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/accelerator.py", line 1373, in prepare_model
    model.forward = MethodType(new_forward, model)
TypeError: first argument must be callable
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1123693) of binary: /root/miniconda3/envs/py3.10/bin/python