Prometheus with Basic Auth

Currently Promlens does not support if Prometheus is protected via basic auth, would be great if that could be added.

+1. Would love to see either Basic Auth working with PromLens or PromLens incorporated into the Prometheus Web UI as an additional option.

Hey, same issue here!
I am hosting PromLens on Kubernetes and I used a workaround this way:

Promlens is hosted in same namespace than Prometheus
I declared two ingress ressources:

One to Promlens service
Another one, on same hostname, which is a proxy to prometheus service. This way, prometheus api is served on same hostname and on prefix: /prometheus

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: http://nginx-ldap-auth.kube-system.svc.cluster.local:5555 # our basic-auth system
  name: promlens
  namespace: monitoring
spec:
  ingressClassName: ingress-private
  rules:
  - host: promlens.myfqdn
    http:
      paths:
      - backend:
          service:
            name: promlens
            port:
              number: 8080
        path: /
        pathType: Prefix

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: http://nginx-ldap-auth.kube-system.svc.cluster.local:5555
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: promlens-to-prometheus
  namespace: monitoring
spec:
  ingressClassName: ingress-private
  rules:
  - host: promlens.myfqdn
    http:
      paths:
      - backend:
          service:
            name: prometheus-prometheus
            port:
              number: 9090
        path: /prometheus(/|$)(.*)
        pathType: Prefix

Thanks for this awesome tool! :)

I just found another nice workaround. Just use --grafana.url. Then Grafana will handle basic auth. PromLens even says this in the logs:
promlens_1  | 2023/01/31 18:27:46 Proxying to Grafana at /api/datasources/proxy/92/api/v1/query...

Cheers :)

I just found another nice workaround. Just use --grafana.url. Then Grafana will handle basic auth. PromLens even says this in the logs:
promlens_1  | 2023/01/31 18:27:46 Proxying to Grafana at /api/datasources/proxy/92/api/v1/query...

Cheers :)

I have a grafana behind TLS but SSL certificate is not valid. Is there a way to disable it ?
tls_skip_verify_insecure maybe ? but where.

I added support for Basic Auth in this Fork (https://github.com/glidea/promlens), and a 'datasource' manager (https://github.com/glidea/promlens-anywhere)

I added support for Basic Auth in this Fork (https://github.com/glidea/promlens), and a 'datasource' manager (https://github.com/glidea/promlens-datasource-manager)

Is there a Docker image that can be used,thanks

I added support for Basic Auth in this Fork (https://github.com/glidea/promlens), and a 'datasource' manager (https://github.com/glidea/promlens-datasource-manager)

Is there a Docker image that can be used,thanks

来兄弟。https://hub.docker.com/repository/docker/glidea/promlens/tags

谢谢兄弟，我试试


Add Tab Ordering and AutoFocus on sign in and Registration Page

Pull request checklist

[ ] My code contains tests covering the code I modified
[x] I linted and tested the project with bin/check
[ ] I added significant changes and product updates to the changelog

This PR address:

[x] Set the tab order on the sign in screen to email then password then forgot password
[ ] Set the tab order on the registration screen to email then password then password confirmation
[x] Autofocus on the email field for Sign in, Registration, Forgot Password and Resend confirmation page

Note: Tab Ordering on registration screen to email then password then password confirmation is already maintained because of its location in source code (these fields are in ascending order as per code) & hence changes are not required
 Screen Recording of Autofocus on mentioned pages
https://user-images.githubusercontent.com/36496890/226115603-188d1edc-8aaf-4357-b4ab-5868cd8bf2a4.mov
 Screen Recording of Tab Ordering on Sign in page
https://user-images.githubusercontent.com/36496890/226115920-3445bc81-261a-48e6-b7ce-cbec1af5ba5e.mov

Hey @aj-abhishek-jadav, is this ready for review? If not, please convert it to a draft PR and mark as ready for review when you'd like me to take a look. Thanks!

Hi @joemasilotti, I m ready with PR. Request you to please Review and let me know if changes are needed.
Btw this is my 1st Open source contribution. Thank you so much for giving this opportunity 😍
Thanks ❤

Hi, I'm sorry to tell you that this change introduces an accessibility issue. The tab order should always be the order in which the elements are defined. If a screen reader user presses tab after the sign in button, he should go to English link, and if he presses shift tab after the field to enter his email, he should go to create an account link. Pressing tab from the top of the page to the end of the page should reflect the logical order. If a user is on the link to create an account and presses tab, he would expect to find the field to enter the email, but instead, he will skip the entire login form and go directly to the English link.
The tabindex attribute should very rarely have values other than 0 or -1. The 0 is used so that elements that are not focusable by the browser by default, such as a div, for example, are focusable by the keyboard and are added to the corresponding tab order. The -1, serves so that an element is keyboard focusable, but is removed from the tab order. This is useful to make a poppup menu, for example, where you want that the elements of the menu are focusable, but that only one of them is in the tab order, the one that is active at that moment.
Actually the solution in this case is that these elements do not have the tabindex attribute at all. If the email field has the autofocus attribute, it will focus automatically. The problem is that when you press tab, it will focus on the forgot password link instead of the password field. The best thing to do in these cases is to change the design, perhaps placing the link in another place that makes sense with the tab order.
I hope this is useful!

Thanks for chiming in! Can you please provide a link showing that breaking "tab order should always be the order in which the elements are defined" is an accessibility issue?

Sure, there is. The second is a rule of axe-core, the best accessibility linter. Also, I tell you this as a blind user and developer.
https://webaim.org/techniques/keyboard/tabindex
https://dequeuniversity.com/rules/axe/4.4/tabindex

Wow, TIL! Thanks for the links and bringing this up.
Would you like to open a PR that removes the tab indexes from the form? No pressure - if not I'll take care of it myself this week.


How force project version on Nexus

Hi
I wrote my gradle configuration follow the example on the README page, but the version in Nexus is like current date (In my settings I used 0.0.1-SNAPSHOT). How can I change this behavior?
This is an extract of my build.gradle file:
version = '1.0-SNAPSHOT'
group = '...'

jar {
    baseName = '...'
    version = '0.0.1-SNAPSHOT'
}

nexus {
    sign = false
    repositoryUrl = '...'
    snapshotRepositoryUrl = '...'
}

Thanks in advance!

That's normal SNAPSHOT behavior in a binary repository. Nexus will need to keep track of all published artifacts by timestamp so it can pick the latest if requested. I believe you can change that setting in the Nexus repository itself but I'd need to look at the options.
BTW: Setting the version property and changing the version for the jar task makes little sense. Why would you want to use two different versions or better "why do you want to overwrite it if it's already set"?

@bmuschko, thanks for your answer. Now I understand why get that versión. So the correct configuration for version in nexus is version = "my-version", outside of jar closure.
Thanks again!


Need substitute for network-profile option to access postgres image from webapp image

Describe the bug
There was a breaking change in 2.29 but no documentation provided regarding how to sustain required functionality.  The line below used to create our container so that it could reach a postgres DB specified by the network profile below.  Can you please point me to some work-around on how to deal with this breaking change?
To Reproduce
az container create -e DB_UserId={dbUser} {dbPswd} DB_SERVER={dbServer} -g rg-{mygroup} --name dev-xxx --image informpredictdev.azurecr.io/ipappweb --ports 80 --registry-username informpredictdev --registry-password xxx --network-profile /subscriptions/{xxx}/resourceGroups/rg-{mygroup}/providers/Microsoft.Network/networkProfiles/testaci-networkProfile
Expected behavior
Would expect some guidance near the "breaking change" documentation to help us upgrade
Environment summary
Azure DevOps
az container create -e DB_UserId={dbUser} {dbPswd} DB_SERVER={dbServer} -g rg-{mygroup} --name dev-xxx --image informpredictdev.azurecr.io/ipappweb --ports 80 --registry-username informpredictdev --registry-password xxx --network-profile /subscriptions/{xxx}/resourceGroups/rg-{mygroup}/providers/Microsoft.Network/networkProfiles/testaci-networkProfile

route to service team

@yonzhan It is related to containers. please route to the containers team.

msftbot will help route to the correct team


Fix (hopefully) uniqueness for gateway AcrPull role assignment

Which issue this PR addresses:
Fixes https://msazure.visualstudio.com/AzureRedHatOpenShift/_workitems/edit/10743419
What this PR does / why we need it:
For some reason one region deployment failed for non-unique GUID in AcrPull role assignment.  In investigating I don't see a great reason why it should fail.  But GUIDs are not guarenteed unique.  This PR simply cleans up the GUID generation as we need to change something for this region.
Test plan for issue:
E2E, verify in INT, verify in Canary.
Is there any documentation that needs to be updated for this PR?
No

GUID in ARM is not unique. Its a static hash based on inputs.

The returned value isn't a random string, but rather the result of a hash function on the parameters. The returned value is 36 characters long. It isn't globally unique. To create a new GUID that isn't based on that hash value of the parameters, use the newGuid function.
https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions-string#guid

This change does not look right. First it changes existing naming patterns (without cleaning old RBAC),
And this:
"name": "[concat(substring(parameters('acrResourceId'), add(lastIndexOf(parameters('acrResourceId'), '/'), 1)), '/', '/Microsoft.Authorization/', guid(concat(parameters('acrResourceId'), parameters('gatewayServicePrincipalId'), 'RP / AcrPull')))]"
vs
"name": "[guid(concat(parameters('acrResourceId'), parameters('gatewayServicePrincipalId'), 'Gateway / AcrPull'))]",
looks like we are removing data, but I dont see how this will make it more unique based on guid behavior.
Second, Error I think is race condition for multiple parallel runs so conflict is coming from there (I think)
we have this:
https://github.com/Azure/ARO-RP/blob/18439dff653218914ab74811c164063cc6dff757/pkg/deploy/predeploy.go#L243-L266
Which deploys this:
https://github.com/Azure/ARO-RP/blob/18439dff653218914ab74811c164063cc6dff757/pkg/deploy/generator/resources_rp.go#L1641
Can it be similar failure mode where same resource is being modified in parallel?

@mjudeikis agree on GUID uniqueness.  Don't agree on removing data.  It stops the parsing of the acrReousrceId to build some mocked up and wrong scope for the guid generation.  It's just wrong, not useful, and complex.  If we want to use scoping (which I don't know how it can be anything other than hashing the input, it is not efficient to check true uniqueness) we might pass subscription as the first argument to guid
On parallel execution we have a single retry.  It's being used.  It is only this one region that is having a problem.  Given that guid generation is not unique I am suspecting we are hitting a case where generation is creating a duplicate.  If we suspect parallel deployments can we stop them from being parallel?  What does it gain us?
On cleanup, yes understood.  This isn't worth the effort to automate cleanup in my opinion.  We have a small number of regions to manually cleanup.  We're already using elevated access to create certs.  This cleanup can be done while waiting for those scripts to run for the next sector.  Or when we run for japaneast as due to the issue this is trying to fix the gateway keyvault is not yet created and we'll need to create certs.

/azp run e2e


Podman action-k3s-helm failing on nektos act

https://dask.discourse.group/t/getting-starting-with-act-locallly-with-podman-need-help-system-has-not-been-booted-with-systemd-as-init-system-p-id-1-cant-operate/2251

"But, afaict the docker that they offer doesn't allow a docker-in-docker to work because it doesn't have --privileged or something (because it doesn't want to be attacked by guest processes)."
https://slack-archive.rancher.com/t/5933919/i-d-like-to-use-nektos-act-a-dockerd-client-on-a-gitlab-runn#:~:text=But%2C afaict the docker that they offer doesn't allow a docker-in-docker to work because it doesn't have --privileged or something (because it doesn't want to be attacked by guest processes).


Removing 2.79

@donmccurdy @emackey @pjoe @scurest
I made a PR for removing 2.79
I think some testing code can be refactored now that there is only 2.80+, but let's start with this PR

Lgtm.
You probably want to bump this line since the addon will no longer work in 2.81.6.

This would be fantastic - makes things so much simpler :)
The CircleCI build should also be quite a bit faster now, looks like it goes from ~5m to 2m - sweetness indeed  🚤
It may be possible to cut buildtime further, by re-distrubuting the remaining tests across 4 runners (which is what Circle allows for OSS) - but that's an optimization that can be done after this is merged.
+1 (or +1000)

New copy tool snuck in at the last minute.  Dare I ask:  Does it handle file deletions?

Yes, it does :)

Wonderful, thank you!


A new "secretMounts" parameter added.

Allows you easily mount a secret as a file inside the deployment. Useful for mounting certificates and other secrets

Guys, is there anything missing here?

@dergraf, @larshesel good to merge or anything missing?


Move browser dom adapter setup to spec.bundle.js

See https://github.com/AngularClass/angular2-webpack-starter/pull/128#discussion-diff-44580177

thanks!


The h30ga model 0.2.2: RSTP stopped working

Tested: default and alternative RTSP servers
The RSTP links are dead - no connection via VLC or other tools
Last working FW: 0.1.9

Please send me info about the processes.
Check if the processes are running:
ps | grep -i rtsp
Check if the port is open:
netstat -an | grep 554
Try the last command even while you are playing the stream.

Please, try the last commit:
https://github.com/roleoroleo/yi-hack-Allwinner-v2/commit/bf96b2413a9512e969afd5a43cc3113c7ddc48e1

Aw man, sorry, I forgot there's ssh console! So, netstat is not present but the process seems to be working when:

I've configured both TWO streams
enabled 'alternative' rtsp server

ps shows now:
rtsp_server_yi -m h30ga -r both -a no -p 554
and camera works back with Shinobi/VLC. No need to upgrade to latest commit!


Add hover feedback styling to diagram statements

Description:
Currently there is only cursor change on mouse hover on an elements which has additional actions. This is not clear enough to identify that they are clickable. We need to have a clear hover state on them.
Suggested Labels (optional):
[Type/UX]

No longer valid since this support has been removed


HTTP 503 -> 500 in core-data service

Refactor core-data API implementation and associated RAML so that all cases where HTTP 503 (Service Unavailable) is returned are replaced with the more appropriate HTTP 500 (Internal Server Error).

Associated with #284


React not working on IE browser

React not working on IE browser

This was resolved in #8439 though we'll have to wait for a release

Waiting for 3.3.2 as well  🙋🏻♂


references are not inherited in tsconfig.json

TypeScript Version:  3.1.0-dev.20180914
Search Terms: tsconfig.json extends references inheritance
It seems that the references key is not inherited via the extends mechanism, which surprised me because the handbook doesn't mention anything special about it.
Demo:
tsconfig.base.json:
{
  "references": [
    { "path": "./some/other/project" }
  ],
  "compilerOptions": {
    "declaration": true,
    "composite": true
  }
}

tsconfig.doesnt-work.json:
{
  "extends": "./tsconfig.base.json"
}

Building tsconfig.doesnt-work.json doesn't build the reference:
$ tsc -b -f -d tsconfig.doesnt-work.json
[11:12:13] A non-dry build would build project 'C:/demo/tsconfig.doesnt-work.json'

tsconfig.works-but-duplicates-references.json:
{
  "extends": "./tsconfig.base.json",
  "references": [
    { "path": "./some/other/project" }
  ],
}

This is a correct build but I had to duplicate the references key:
$ tsc -b -f -d tsconfig.works-but-duplicates-references.json
[11:12:13] A non-dry build would build project 'C:/demo/tsconfig.works-but-duplicates-references.json'
[11:12:13] A non-dry build would build project 'C:/demo/some/other/project/tsconfig.json'

This is the intended behavior but we need to document it

I tried to use inheritance of references section to setup a second tsconfig.json for production builds (without sourcemaps etc.). Since there is no inheritance I need to duplicate the refrences section. Perahps I'm not doing the right solution for this scenario? How would you handle a separate production build with project references?

@RyanCavanaugh, still curious about the reasoning behind this design decision.
I have a use case where I need to switch the module system from es6 to commonjs for my karma-typescript tests. This is easily done by extending the original tsconfig, changing this option and referencing the extended tsconfig in the karma config. But by doing so, all the project references are lost.
While there may be reasons for not wanting to extend this attribute, I think there are also legitimate for doing so. Copying the references is error-prone, people forget about it. Maybe you can find a way to allow this in a future version. Thanks :)

For some hierarchy
A
⬆
B
⬆
C

B could be, for example:

An opaque library that doesn't intend to expose its reliance on A to its consumer C and leaks no types from A
A helper library that adds additional A-related functionality that C might want to use

If references are inherited, then the first scenario is impossible to do.
If they're not inherited, then you can pick which you mean from C's perspective.

Got caught by this. In my setup I have: tsconfig.json and tsconfig.build.json per package. the first is for testing (via mocha + ts-node) and the second is derived from the first, but altered for the distribution build. Without inheritance of references I now have to duplicate them in both.

@RyanCavanaugh Rather than having an extra tag in references, why not optionally allow references to be placed into a separate json file and then be refer to it in the config?
That way cases where people specifically want to list their references in a single location and share those references between two or more tsconfig.json files, they can do so without impacting those that might be affected the proper functioning of your first scenario above?

@RyanCavanaugh Thanks for explanation, but I'm not sure if expected behavior and explanation provided aligns with common understanding of what "extends" mean in scope of configuration.
The example you've provided make perfect sense if A, B, C would be libraries and edges are use relations, which should be private and that's how it works in proper dependency resolution systems.
Whereas, when it comes to configuration - at least pople in this thread -  inheritance means "just" take all the props from parent config, so basic "public" inheritance like we experience with Typescript extends and/or _.merge.
So, summarizing design you're referring IMHO conflicts with common understanding of inheritance in configuration systems.
Is there chance to have workaround? (given that - assuming from your comment - current behavior is already set in stone).

I support the opinion, that current inheritance model isn't optimal.
Because tsconfig.json is a plain json file (very pity!), there is no sophisticated way to workaround it. Just for simply cases for monorepos I would like to promote my alpha set-project-references helper. It doesn't help with inheritance, but makes life with project references easier.

^^ Documented this as it bit me hard tonight.

In the handbook it says:

files, include and exclude from the inheriting config file overwrite those from the base config file.

Following the principle of lease surprise, references should also inherit and overwrite those from the base config file.
Regarding what Ryan said:

An opaque library that doesn't intend to expose its reliance on A to its consumer C and leaks no types from A
...
If references are inherited, then the first scenario is impossible to do.

Is it still impossible if we allow references to override even if it's empty?
{
    "extends": "./tsconfig-other.json",
    "references": [],
}

I vote for inheritance.

Surprised++
The hierarchy/hiding argument seems a bit weak to me. Let's make those edge cases not use inheritance and just copypasta rather than changing what inheritance means structurally.
I am also trying to create a production build and avoid to compiling tests files that depend on test packages. If I exclude test files from my normal tsconfig that breaks other things (like IDEs).

TSC should always tell me if something like this is wrong. Please don't assume everyone reads the documentation right away. Becuase referencing another project somehow worked on me because of my "paths" compiler options, so I had NO idea the actual project references of my BASE tsconfig was not working. If TSC sees something fishy please ALWAYS tell the user

I'll jump in this and bump this issue. I have a tsconfig.build.json to exclude test files from builds and expected references to inherit as this isn't clearly documented.
It isn't particularly optimal to duplicate the references in tsconfig.json and tsconfig.build.json but seems like that's the only working solution currently?

this is my unfortunate workaround for this issue:
A script that copies tsconfig, adds the tests folder into the exclude list and runs tsc on this new config file
It would be really nice to get a real solution for this.
const path = require("path");
const fs = require("fs");
const spawn = require("child_process").spawn;

const tsConfigPath = path.join(process.cwd(), "tsconfig.json");
const tsConfig = require(tsConfigPath);
tsConfig.exclude = tsConfig.exclude ?? [];
tsConfig.exclude.push("src/test");

const prodTsPath = "tsconfig.prod.json";

fs.writeFileSync(prodTsPath, JSON.stringify(tsConfig, null, 2));

const tscProcess = spawn("tsc", ["--build", prodTsPath], { cwd: process.cwd() });
tscProcess.stdout.pipe(process.stdout);
tscProcess.stderr.pipe(process.stderr);

tscProcess.on("close", (code) => {
  fs.rmSync(prodTsPath);

  setImmediate(process.exit(code));
});

seems to be best monorepo setup I have found to date for typescript.  This would be amazing if fixed.  Reference blog that refers to this bug -> https://blog.ah.technology/a-guide-through-the-wild-wild-west-of-setting-up-a-mono-repo-with-typescript-lerna-and-yarn-ed6a1e5467a

This one just bit me hard. And since it has been this way for years there's no way to roll it back in. This is a real bummer because separate build/test/lint configs are more necessary than not, and generally they all extend a base config.
Maybe the "extends" property can accept an object with an includeReferences property which defaults to false to match the current behavior. This could also work with the new array format for extends:
{
  "extends": {
    "path": "../path/to/project",
    "includeReferences": true
  }
}

I have dozens of packages in my monorepo, and I want to exclude test files from my production builds. So I need dozens of tsconfig.build.json files alongside my tsconfig.json files to specify the excluded tests -- but now I have to specify those dozens of lists of project references twice. This design pretty well guarantees I will trip myself up over and over.
Would definitely like to understand better why the references can't be inherited ...

I think this is dumb. Why have references at all! Instead just look at the node_modules folders and go up the tree like node does. Then all these workspaces would just work without having to manually add references or even worse add some huge library like Nx to manage it all. It is like they want to make it as complex as possible.

Oh god... This should be implemented.. Any update on this is it considered or not?


TAYGA LD27 TO BE DONE WORK HARDESSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT TASK TAYGA

01'TAYGAECOWAR
27 TAYGA LD27 TO BE DONE WORK HARDESSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT TASK TAYGA

https://github.com/FUTUREDESIGNDEV/HUTORI2PCURRENTAGENDA/issues/32


Adding the cheriot-audit tests to the test suite

This PR adds the test suite from https://github.com/CHERIoT-Platform/cheriot-audit/ to this repo.

Anything in particular you would like me to look at in a review?

I'm talking with David about options. I think we're going to change the tactic to include his repo and and build it against the current repo/branch similar to how we do with Python and Rust, instead of manually copying it in as I've done. He needs to do a bit of instrumentation first, but that way we don't have duplication.

CHERIoT-Platform/cheriot-audit#4 should make that option easier.

This has been done via another means.


favicon

favicon
Assign this to me

Hi @joyhughes Naunihal Singh this side from Gssoc, favicon addition is a great task to do.
I request you to assign this task to me.
I will be glad to have this opportunity!


Add hover effect to buttons in the "Why Choose Us" section for improved UI

I suggest adding a hover effect to the buttons in the 'Why Choose Us' section to enhance the user experience and make the UI more engaging. A subtle hover effect will highlight each button as the user interacts with it, providing a more polished and interactive feel. Let me know if this would be a good enhancement!

yes its also missing out there

Thanks for creating the issue in WildGuard! The issue has been assigned to you.
Before you start working on your PR, please make sure to:

⭐ Star the repository if you haven't already.
Pull the latest changes to avoid any merge conflicts.
Attach before & after screenshots in your PR for clarity.
Include the issue number in your PR description for better tracking.

Don't forget to follow @anuragverma108 – Project Admin & @rahulkhandait-sde - Project Collaborator  – for more updates!
We're excited to see your contribution as part of Hacktoberfest 2024! 🎉
Happy open-source contributing!


Add ubuntu 18.04, 20.04 and debian 10 support

Pull Request (PR) description
Add ubuntu 18.04, 20.04 and debian 10 support

@olifre if you could take a look.

@jblomer there is no cvmfs-config repo for debian ?

@traylenator That looks good from looking through the code!
Since our infrastructure is not yet on Puppet 6 (@wiene and I are still working on the upgrade), I can not test it on real machines (yet).
One thing I am not sure about is whether people may want to be able to configure
ensure  => refreshed
for the key of the apt::source.
I personally believe there is no security risk involved in refreshing, since puppet-apt only does it when the key is actually expired, using existing keyserver infrastructure (see https://github.com/puppetlabs/puppetlabs-apt/blob/8927af46c0bb430be677bc919323c133204a253d/lib/puppet/provider/apt_key/apt_key.rb#L179 ), but it rather saves a lot of hassle in real life. It has saved us in several cases for many repositories when they prolong their keys, which would be either an additional Puppet hack or manual work.

@olifre should still work on puppet 5 , only removed really as no puppet5 avilable for 20.04.

ensure => refreshed on key.

Seems like a bad idea to me, if someone wants they can submit a patch.

@olifre should still work on puppet 5 , only removed really as no puppet5 avilable for 20.04.

@traylenator Indeed, there does not seem to be anything in the code requiring Puppet 6 ;-).
I tested this on a Debian 10 node just now, and it works fine — you also changed the repo URI to https, thanks! :+1:


[Dialogic] Saved definitions not present, creating file: 0

[Dialogic] Saved definitions not present, creating file: 0
i just installed this plugin and i get this error when i play a dialogue
it prints [Dialogic] Saved definitions not present, creating file: 0  in debug
Expected behavior
it should play the dialogue
System (please complete the following information):

OS: Windows
Godot Version:  3.3
Dialogic Version: e.g. 1.1

@pranshuthegamer can you try with the current 1.2 version from main?

@pranshuthegamer can you try with the current 1.2 version from main?

so you want me to download main?

ok it works now


pycolmap has moved into colmap/colmap

Comment:
As described in colmap/pycolmap it has migrated to the main repo, colmap/colmap; should we worry about the package outdated, and try to move the source of pycolmap to colmap?

Yes please - are you happy to give it a go @HernandoR?

Sure,  I will give it a try, but I have never written a conda recipe before, so you may need to pay extra attention to review my pr introducing serious bugs.

@Tobias-Fischer Do you mind check out on https://github.com/HernandoR/pycolmap-feedstock/issues/1 ?
i'm totally lost, don't know where referance to or what to do on next step.

Can you please create a pull request here in the repository? Then we can debug here and see what’s going on in CI.

All done :)


Add GitHub reviews status when merging

Q
A

Bug fix?
no

New feature?
yes

BC breaks?
no

Deprecations?
no

Tickets
Fix #20

License
MIT

$ hk merge 138
Working on hupkit/hupkit (branch 1.x)

 ! [NOTE] Using local configuration from branch "_hubkit".

Merging Pull Request 138: Add GitHub reviews status when merging
https://github.com/hupkit/hupkit/pull/138

 [WARNING] Status checks are pending, merge with caution.

---------------------------------------------------
  Item                  Status     Details
---------------------------------------------------
  PhpStan (8.2)         FAIL
  PHP CS Fixer (8.2)    OK
  PHPUnit (8.1)         OK
  PHPUnit (8.2)         OK
  PHPUnit (8.3)         OK
  Reviewed              Pending    No reviews yet
---------------------------------------------------

 [WARNING] One or more status checks did not complete or failed. Merge with caution.

 1.x branch is aliased as 1.2-dev (detected by composer.json "extra.branch-alias.dev-1.x")

 Category (feature, refactor, bug, minor, style, security)? [feature]
 >

And just now noticed the branch-alias is not up-to-date 😅


[Bug] Invalid source

1025114203545141313/"moxiethespecialist"#0: Invalid source:<PHONE_NUMBER>545141313
I'm trying to get it to send files directly to my dms but it won't do it, everytime I try, this shows up when using my channel ID, I don't really understand why this happens since it was working before

nvm I just forgot how this worked


[Dataflow] Fix errors in README.md

Short PR to fix errors in the README.md.

@jabubake can I merge this?


prometheus-prefect-exporter helm chart

HI,
I'm using Prefect Platform everyday for our ML Engineers. First of all, thanks for spend time on improve Prefect and maintain helm charts.
In our case, we monitoring Prefect with Prometheus and visualizate metrics with Grafana, so we develop a simple exporter written in Python (currently migrate to Golang), but for now, works properly and provide us enough metrics to generate alerts if some flow-run failed, all deployments are paused, if work queues or work pools aren't available...
So, here are the repositories:

prometheus-prefect-exporter: https://github.com/devops-ia/prometheus-prefect-exporter (written in Python) but I'm working to rewrite on Golang. Here's Docker releases: docker releases
helm chart prometheus-prefect-exporter: https://github.com/devops-ia/helm-charts/tree/main/charts/prometheus-prefect-exporter
dashboards: prefect-platform-overview | prefect-platform-flow-run-overview

If you want, I can transferring a repository and open new PR on your new transfered repository. So, that's all... if you want speak or next-steps, I will be waiting. Thank you very much for everything.

yes @jimid27, my proposal:

transfer repo prometheus-prefect-exporter to PrefectHQ org. I can do it now.
create your own helm-chart repo and move only code chart from my repo (because I'm using same repository for differents chats I'm not isolated charts per project). If you want I can open a PR with all code when you've the repository created.

When you confirm I'll start the transfer and wait until you create helm-chart repository.
Thx!

This plan works for me!

@jimid27 i couldn't transfer repo because i haven't permissions to create on your repo (is normal i've already read github's docs). So, you should fork prometheus-prefect-exporter repo on your org. Sorry :(

Hey @ialejandro! Would it be possible for you to give my GH user (@jimid27) permission to the repo so I can initiate the transfer?  I should have those required permissions.
Also if you would like to DM about this I am available on the Prefect Community Slack! (Jimi Dodd-o)


gitlab-runner: 15.4.0 -> 15.5.0

Description of changes
https://gitlab.com/gitlab-org/gitlab-runner/blob/v15.5.0/CHANGELOG.md

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 197352 run on x86_64-linux 1

1 package built:

gitlab-runner

thanks!


chore: added dependabot configuration

What this PR does / why we need it:
Added a dependabot config file to update dependencies
Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #2225
Checklist:

[ ] Docs included if any changes are user facing

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>5:
0.0%

Covered Lines:
85

Relevant Lines:
85

💛  - Coveralls

See https://github.com/kubeflow/training-operator/issues/2225#issuecomment-2293568554.
@tenzen-y do you still want to hold on dependabot integration until v1 api is removed?

See #2225 (comment).
@tenzen-y do you still want to hold on dependabot integration until v1 api is removed?

Yes, I want to block this until v1 removal. Just FYI, we will release the final v1 API version within this year or early Jan.
After that, we will remove the v1 API from the master branch (the v1 is still supported in the release branch).
/hold


fix for #56

@eschreiner
I'm closing this PR without merge. Please, feel free to reopen it if the suggested solution is not good enough for your case.


Timeseries segmentation using unsupervised data?

Hi, I enjoyed reading your paper and understood (to some extent) how it works on supervised data. I have seen couple of other research papers using UNet for unsupervised segmentation of time series data. Are you also planning to implement unsupervised segmentation using your network architecture?

Hi sangramkapre! Sorry for my very late response. We do not currently have plans to use U-Time in the unsupervised setting, but I would be interested to read some of the papers you mention. Please feel free to forward any that you find interesting to me. Thanks!


Service links aren't being created in Bridge mode

I recently upgrade from rancher server v1.9.x to v1.0.0. The upgrade wiped out my Rancher DB server so I had to start fresh. (Clean DB, no previous state).
Service links don't get created after upgrading to v1.0.0 which prevents any inter-container communication from happening when I use Bridge networking.
Rancher Version: v1.0.0
Docker Version: v1.6.2
OS: Ubuntu Server 3.13.0-83-generic # 127-Ubuntu SMP
Steps to Reproduce:

Create a new service stack: (Sample from a production .yml)

redis:
  cap_drop:
  - ALL
  labels:
    io.rancher.container.pull_image: always
  image: redis:3.0.7-alpine
  net: bridge
  mem_limit: 268435456
  cap_add:
  - CHOWN
  - SETGID
  - SETUID
WebUI:
  cap_drop:
  - ALL
  tty: true
  image: ubuntu:latest
  links:
  - redis:redis
  stdin_open: true
  net: bridge

Login to the WebUI container with "Execute Shell"
Execute cat /etc/hosts or ping redis

Results:
The container 'redis' isn't resolvable
sudo docker inspect r-TestStack_WebUI_1:
[...]
    "HostConfig": {
[...]
        "IpcMode": "",
        "Links": null,
        "LogConfig": {
            "Config": null,
            "Type": "json-file"
        },
[...]

Redis NetworkSettings:
    "NetworkSettings": {
        "Bridge": "docker0",
        "Gateway": "<IP_ADDRESS>",
        "GlobalIPv6Address": "[---]:beef:231:359e:ae61",
        "GlobalIPv6PrefixLen": 80,
        "IPAddress": "<IP_ADDRESS>",
        "IPPrefixLen": 16,
        "IPv6Gateway": "fe80::1",
        "LinkLocalIPv6Address": "fe80::31:35ff:fe9e:ae61",
        "LinkLocalIPv6PrefixLen": 64,
        "MacAddress": "02:31:35:9e:ae:61",
        "PortMapping": null,
        "Ports": {
            "6379/tcp": null
        }
    },

WebUI NetworkSettings:
    "NetworkSettings": {
        "Bridge": "docker0",
        "Gateway": "<IP_ADDRESS>",
        "GlobalIPv6Address": "[---]:beef:231:351c:63a8",
        "GlobalIPv6PrefixLen": 80,
        "IPAddress": "<IP_ADDRESS>",
        "IPPrefixLen": 16,
        "IPv6Gateway": "fe80::1",
        "LinkLocalIPv6Address": "fe80::31:35ff:fe1c:63a8",
        "LinkLocalIPv6PrefixLen": 64,
        "MacAddress": "02:31:35:1c:63:a8",
        "PortMapping": null,
        "Ports": {}
    },

Expected:
I expect that the redis container is available as a link to the WebUI container. This is how it worked previously before I upgraded to Rancher v1.0.0. So far, no links appear to be created for any of my services in any of my stacks.
I've tried deleting and recreating the stack several times to see if that fixes anything, but so far nothing.

Can you clarify some things on your versioning?
What was your previous Rancher version? You mention v1.9.x, but everything before was v0.63.1 or v0.59.1, nothing close to v1.9.x.
And are you saying your Docker version is actually 1.6.2? If so, we recommend upgrading to Docker 1.10.3 as that's what we do all of our testing.
As for your specific issue, can you check that cross host communication is working. Can you exec into the network agent on 1 host and ping the IP (10.42.x.x) of another network agent? Also, you say that Rancher's upgrade wiped out your state so you had to start fresh, but you mention an upgraded setup. Is it a fresh install using the built in mysql server or where is the DB located?

You're probably right. I'm not totally sure what the previous version was since I don't have the server container around on my host anymore. It would have been either v0.59.x or v0.63.x.
Basically doing an in-place upgrade keeping the Rancher server DB data around failed (This used to be an option that was mentioned in the upgrade docs) so I just deleted everything and started from scratch. That means brand new Rancher (no previous DB state) and wiped away all of the Rancher agent state data.
I'll see about upgrading Docker to 1.10.3 in a few minutes to try that out.
As of now, there is only one Rancher agent on this server so I can't ping any other agents and actually test anything useful since this is just a test environment.
Also I'm currently testing with the Bridge network configuration, not the Managed network configuration. Which as I understood it, did not use the Rancher network agent. I'm trying to start simple first.
From what I can tell, Rancher doesn't request that Docker create a container Link, which I expected since this was the Docker-based network connectivity.
tl;dr Fresh Rancher everything, single agent Rancher cluster, Non-Managed network configuration. Will try upgrading Docker to try that.

After an eventful upgrade to 1.10, I had to again recreate my Rancher configuration + all containers. It appears to be generating the service Links correctly for Bridge networks.
I'm no-longer impacted by this issue, as such am going to resolve this issue. Thanks for your help and much appreciation on Rancher.


C# testdata returns FSharpList

Describe the bug
When requesting a list of random Swedish PersonalIdentity Numbers (SwedishPersonalIdentityNumberTestData.GetRandom(x)) it returns a FSharpList (Microsoft.FSharp.Collections.FSharpList), even when consumed from the C# facade.
What area is it related to
SwedishPersonalIdentityNumberTestData
Expected behavior
Return an IEnumerable as the rest of the methods does.
Actual behavior
Returns Microsoft.FSharp.Collections.FSharpList.
Screenshots
If applicable, add screenshots to help explain your problem.

NuGet package version
1.0.0
Runtime version
.NET Core 3.1.0

Yes this is clearly a bug.
Can we fix this in the next release or should we do a hotfix just for this?

Can we fix this in the next release or should we do a hotfix just for this?

It's technically a breaking change, so we should wrap this up in the 3.0.0 release.


Update verify.pipeline.sh

Description
Security test
Related Issue

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Chore (non-breaking change that does not add functionality or fix an issue)

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I have run the pre-merge tests locally and they pass.
[x] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.
[x] All commits have been signed-off for the Developer Certificate of Origin.

Hey Chef team, I'm sending you an email with all the details to<EMAIL_ADDRESS>


Add test for setting an empty field's first item to null

Spinning from https://git.drupalcode.org/project/drupal/-/merge_requests/454#note_55009
Drupal.org issue: https://www.drupal.org/project/drupal/issues/3254966
-    $entity->field_test[0] = NULL;
+     $entity->field_test = NULL;

Without this fix, PHPStan crashes when running against Drupal core
Internal error: Internal error: Cannot assign an empty string to a string
offset in file
/var/www/html/core/modules/link/tests/src/Kernel/LinkItemTest.php
Run PHPStan with --debug option and post the stack trace to:
https://github.com/phpstan/phpstan/issues/new?template=Bug_report.md
Child process error (exit code 1):

It'd be great to analyze core/modules/link/tests/src/Kernel/LinkItemTest.php and test the EntityFieldsViaMagicReflectionExtension return type extension.
Full code example to test:
$entity = EntityTest::create();
$entity->field_test = ['uri' => 'internal:/node/add'];
$entity->field_test = [
  'uri' => 'internal:/node/add',
  'options' => ['query' => NULL],
];
$entity->field_test = 'internal:/node/add';
$entity->field_test[0] = NULL;

This seems to be fixed in the lastest versions of phpstan / phpstan-drupal. So this issue can pobably be marked as fixed.

Woo! thanks @mallezie.


Error in webhook of workflow and task statuses

Describe the bug
When driver is configured without these 2 properties:

conductor.status-notifier.notification.headerPrefer
conductor.status-notifier.notification.headerPreferValue

The request is not a valid http
Details
Conductor version: 3.19.0
To Reproduce
Steps to reproduce the behavior:

Configure conductor withtou conductor.status-notifier.notification.headerPrefer and conductor.status-notifier.notification.headerPreferValue
Create workflow
Start workflow
See request sended by webhook is http invalid

Expected behavior
You should check that they have content before doing
headers.put(config.getHeaderPrefer(), config.getHeaderPreferValue());

Thank you


feature request: add error return to ValidateAny()

ValidateAny returns a nil instance if no conforming schema is found but no error, so you end up with this perfectly-valid-but-not-very-go-ish code block:
i := lin.ValidateAny(dashdata)
if i == nil {
  fmt.Errorf("BADONG") //  if you get this ref we are friends
}

It would be nice to have something more idiomatic:
i, err := lin.ValidateAny(dashdata)
if err != nil {
  return err
}

I think this would be nice to have even if we don't do anything to polish the returned errors (yet 😉)

i had a similar thought when i initially wrote it. But i also had a conundrum: in a multi-schema lineage where all schemas fail validation, which error do we return? In what order? For that matter, in what order should ValidateAny() move through all the schemas in a lineage?
Now that we're further along and starting to think more properly about errors, i don't actually find the above questions as intimidating: some kind of multierror that contains all of the schemas' error responses, keyed by version. (If we see other use cases for "error across all schemas," we could later expose a type for this.) And then yes, put that all into an additional error return value.
That said, ValidateAny() has always been a bit of a hack. It's fully implementable externally, so is redundant bloat on the Lineage interface. It was a shortcut that let me create vmuxers without having to design some more proper indirection patterns to, at minimum, allow the caller control over iteration order.
So: i'm :+1: to this change and would happily accept a PR, but would like to see it moved into a standalone func. An initial PR could kick the can down the road on iteration order - adding an error return and moving the func is sufficient. i suspect we can use a variadic option pattern later to avoid another breaking change.


Fix issues with natsjskv registry

Fixes TTL and Registration interval issues that lead to increased memory consumption
Fixes https://github.com/owncloud/ocis/issues/9650

@kobergj I tried to clarify the naming of the TTL variable.

Sure. Just go for it :+1:

hm, I thinrk the natsjservice registy still has a bug. reregistering should not create a new key when the same node registers itself. 👀 🤔

It should. The random part of the key is created on Register. Hence one services will be registered twice until it expires.

I see that it does, but IMO the same service re-registering itself should not get a new uuid ... the id is created when the service is started as part of the node metadata. the node metadata already contains an id, eg. com.owncloud.api.settings-51921b78-e6d6-4866-8eda-7060b3eae3f4. that id does not change. which is correct IMO. if we write the same key in nats it will reset the ttl.

I see your point. But this would require touching ALL services. In some passing options to the registry is not trivial. It would require a major refactor.


[Site] faq title arrows should be spaced apart

Current Behavior
The meshery nighthawk faq title and arrow are adjacent to each other
Expected Behavior
It should be space apart and consistent irrespective to the faq title length
Screenshots/Logs

Contributor Resources

See the Contributing to getnighthawk.dev Website

🎨 Wireframes and designs for Nighthawk Site in Figma

See the Project Site section of GetNighthawk project doc
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

Please assign this to me

@akshitarora921 go ahead !! 🚀


400 Bad Request: The CSRF token is missing

Hi, is this repo still actively maintained. I have been testing this on a Docker deployment of Superset and unable to disable CSRF tokens. there are a number of threads on this which seem to indicate this is a common issue. Is there a way to ensure this library works for POST commands with SS or any specific steps to disable CSRF.

Works after edit of config py vs  Docker env file edit.  Note that when pushing descriptions to latest ss the hover over Works but there is no info icon.  Closing issue for now.


Implement custom layout for head to fix page description

As discussed in #124 revert pull #127 and implement a custom file in the /layouts folder to solve the requirement.

Please don't revert me! I'm ok!

Haha, sorry! Should have used a # instead of a @. Don't worry we wont be reverting you 😂


stop relying on HF token

store model in real S3 bucket, centrally
sync model into minio?
deploy from bucket, not HF

or use modelcar concepts


Bias support in TransposeConv2d for TFLite Delegate

Hey,
I would like to request support for bias in TransposeConv2d (https://github.com/ARM-software/armnn/blob/branches/armnn_22_05/delegate/src/Convolution.hpp#L563). Currently it is suported for Conv3d or DepthwiseConv2d. I guess the implementation could be the same as in Depthwise variant, but I am not experienced enough with ArmNN or implementing neural network operators to evaluate that.

Hello @MrSherish,
I've added a ticket for adding the operator to our backlog however there is no ETA at the moment when they will be implemented. As we're going to track these internally, I'm going to close this issue but we'll update it if and when support for these operators have been added.
If you would like to help implement these operators then here is a link to our contributors guide which will show you how you can submit a patch to us. If you have any questions then please feel free to create a new Issue, thank you!
https://www.mlplatform.org/contributing
Kind Regards, Cathal.


Simplify language and structure for clarity and ease of use.

Summary of the Pull Request
Streamlined language, clarified sections, and enhanced instructions for a more concise, user-friendly contributor guide.

PR Checklist

[x] Closes: #xxx
[x] Communication: I've discussed this with core contributors already. If work hasn't been agreed, this work might be rejected
[x] Tests: Added/updated and all pass
[x] Localization: All end user facing strings can be localized
[x] Dev docs: Added/updated
[x] New binaries: Added on the required places

[x] JSON for signing for new binaries
[x] WXS for installer for new binaries and localization folder
[x] YML for CI pipeline for new test projects
[x] YML for signed pipeline

[x] Documentation updated: If checked, please file a pull request on our docs repo and link it here: #xxx

Detailed Description of the Pull Request / Additional comments

Headings for Clarity:

Created a clearer heading for the “Filing an Issue” section to emphasize its importance.
Changed "How to Tell the PowerToys Team This is an Interesting Thing to Focus On" to “Indicating Interest in Issues” for conciseness.

Revised Filing Instructions:

Broke down filing instructions into concise, actionable points.
Clarified when and why users should file issues and conduct a quick search before filing.

Simplified "Contributing Fixes/Features" Section:

Added clear instructions to comment on interest before starting work.
Reworded to avoid redundancy, especially around the team’s guidance and multiple contributors.

Streamlined Localization and Help Wanted Sections:

Rephrased the localization section to be direct, avoiding unnecessary detail.
Revised “Help Wanted” to direct users simply to the labeled issues for easy navigation.

Refined "Thank You" Note:

Added a closing thank-you note that highlights collaboration and the value of community contributions.

These changes make the guide easier to follow, more structured, and clearer for contributors.

Validation Steps Performed
All tests passed, confirming feature reliability.

@microsoft-github-policy-service agree

Good job!

Thanks! Glad it helped!


Implementation for LCD-TFT display controller

Somewhat replicates the implementation from the F7 HAL, but with a re-usable traits that should apply to a wide range of displays and at least stm32 microcontrollers.

This would be a good candidate to be shared between f4/f7/h7, as well as the FMC

Reopened after it was closed by an errant bors

Reopened after it was closed by an errant bors

bors r+


fixes: should no defer when detach

For this PR: #2798 , add flag -d for ctr t exec to run a service in container.
There is a small error in Line 97: https://github.com/lifubang/containerd/blob/01f5aa3878cf153c36fcc1dcd3789d4e229c835f/cmd/ctr/commands/tasks/exec.go#L93-L99 , because when we use detach, we should not call defer process.Delete(ctx). This is my fault.
But there is no error for ctr, because we don't print the error for process.Delete(ctx). If we print, we will get an error: process must be stopped before deletion: failed precondition.
So, if this is not import, feel free to close this PR.
Signed-off-by: Lifubang<EMAIL_ADDRESS>

@lifubang do you mind to repush the commit to rerun the CI? The failed case seems like flaky case.

Codecov Report

Merging #2851 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2851      +/-   ##
==========================================
- Coverage   43.78%   43.75%   -0.03%
==========================================
  Files         100      100
  Lines       10751    10751
==========================================
- Hits         4707     4704       -3
- Misses       5315     5317       +2
- Partials      729      730       +1

Flag
Coverage Δ

#linux
47.41% <ø> (ø)
:arrow_up:

#windows
40.91% <ø> (-0.04%)
:arrow_down:

Impacted Files
Coverage Δ

gc/scheduler/scheduler.go
67.3% <0%> (-1.45%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09a5b1f...2d3a4f9. Read the comment docs.

LGTM


When kserve releases a new version of the model, the current version is 9 and the new version is 10. Why does version 8 start ？

/kind bug
What steps did you take and what happened:
Deploy a model which version in knative is 9. Before that version 8 has been deployed successfully.
When I deploy version 10 with canary percent 1% 。As a result pods in version 8 and version 10 has been created.
What did you expect to happen:
Pods in version 10 are created only.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

Istio Version: 1.11.4
Knative Version: 1.0.0
KFServing Version:
Kubeflow version:
Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
Minikube version:
Kubernetes version: (use kubectl version): v1.21.3
OS (e.g. from /etc/os-release): centos 7

@wenyangchou Was version 9 rolled out with 100 percent traffic?

@wenyangchou Was version 9 rolled out with 100 percent traffic?

Version 9 was rolled out with 100% , and version 10 was rolled out with 1%

in that case version 9 and 10 should have the split of 90/10%, which version you are on?

The latest version is 9 and it has been rolled out with 100 percent traffic. Now I want to roll out a new version 10 with 1 percent traffic. At this time, the previous version 8, which pod has been  already terminated was started. As a result, there are three version with their pods in running status at the same time.

@wenyangchou can you provide the inference service status details? There can be only two versions running at the same time.

@yuzisun
Command:
kc get pod -n phantom |grep resort

Shows:
resort-predictor-default-00005-deployment-7f5c9b6674-444ks        1/2     Terminating   0          24s
resort-predictor-default-00005-deployment-7f5c9b6674-z7sxw        2/2     Running       0          24s
resort-predictor-default-00006-deployment-ffb6fc6df-j8fql         2/2     Running       0          3m22s
resort-predictor-default-00006-deployment-ffb6fc6df-xg8xz         2/2     Running       0          3m14s
resort-predictor-default-00007-deployment-8cfff6d95-2wn4j         2/2     Running       0          23s
resort-predictor-default-00007-deployment-8cfff6d95-svpwb         2/2     Running       0          23s

Then:
kc get isvc -n phantom|grep resort

shows:
resort                     http://resort.phantom.dev-predict-phantom.amh-group.com                     True    99     1        resort-predictor-default-00006                     resort-predictor-default-00007                     10d

Inferenceservice details shows:
Name:         resort
Namespace:    phantom
Labels:       predict-phantom.amh-group.com/modelName=resort
              predict-phantom.amh-group.com/modelVersion=v2.1
Annotations:  predict-phantom.amh-group.com/group:
              predict-phantom.amh-group.com/message:
              predict-phantom.amh-group.com/modelName: resort
              predict-phantom.amh-group.com/modelVersion: v2.1
API Version:  serving.kserve.io/v1beta1
Kind:         InferenceService
Metadata:
  Creation Timestamp:  2021-12-30T08:40:55Z
  Finalizers:
    inferenceservice.finalizers
  Generation:  12
  Managed Fields:
    API Version:  serving.kserve.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:predict-phantom.amh-group.com/group:
          f:predict-phantom.amh-group.com/message:
          f:predict-phantom.amh-group.com/modelName:
          f:predict-phantom.amh-group.com/modelVersion:
        f:labels:
          .:
          f:predict-phantom.amh-group.com/modelName:
          f:predict-phantom.amh-group.com/modelVersion:
      f:spec:
        .:
        f:predictor:
          .:
          f:canaryTrafficPercent:
          f:maxReplicas:
          f:minReplicas:
          f:nodeSelector:
            .:
            f:device:
    Manager:      okhttp
    Operation:    Update
    Time:         2022-01-10T03:33:14Z
    API Version:  serving.kserve.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:spec:
        f:predictor:
          f:containers:
      f:status:
        f:components:
          f:predictor:
            f:latestCreatedRevision:
            f:latestReadyRevision:
            f:traffic:
        f:conditions:
    Manager:         manager
    Operation:       Update
    Time:            2022-01-10T03:33:23Z
  Resource Version:  129932078
  UID:               90ef21b0-6777-4d0f-be31-c46dd74e32d0
Spec:
  Predictor:
    Canary Traffic Percent:  1
    Containers:
      Env:
        Name:             PHANTOM_STORAGE_URI
        Value:            http://local
        Name:             PHANTOM_MODEL_NAME
        Value:            resort
        Name:             PHANTOM_MODEL_VERSION
        Value:            v2.1
      Image:              harbor.ymmoa.com/matrix/orochi/resort:v2.1
      Image Pull Policy:  Always
      Name:               kserve-container
      Resources:
        Limits:
          Cpu:     2
          Memory:  4Gi
        Requests:
          Cpu:     2
          Memory:  4Gi
    Max Replicas:  2
    Min Replicas:  2
    Node Selector:
      Device:  cpu
Status:
  Address:
    URL:  http://resort.phantom.svc.cluster.local/v1/models/resort:predict
  Components:
    Predictor:
      Address:
        URL:                        http://resort-predictor-default.phantom.svc.cluster.local
      Latest Created Revision:      resort-predictor-default-00007
      Latest Ready Revision:        resort-predictor-default-00007
      Latest Rolledout Revision:    resort-predictor-default-00006
      Previous Rolledout Revision:  resort-predictor-default-00005
      Traffic:
        Latest Revision:  true
        Percent:          1
        Revision Name:    resort-predictor-default-00007
        Latest Revision:  false
        Percent:          99
        Revision Name:    resort-predictor-default-00006
        Tag:              prev
        URL:              http://prev-resort-predictor-default.phantom.dev-predict-phantom.amh-group.com
      URL:                http://resort-predictor-default.phantom.dev-predict-phantom.amh-group.com
  Conditions:
    Last Transition Time:  2022-01-10T03:33:23Z
    Status:                True
    Type:                  IngressReady
    Last Transition Time:  2022-01-10T03:33:23Z
    Severity:              Info
    Status:                True
    Type:                  PredictorConfigurationReady
    Last Transition Time:  2022-01-10T03:33:23Z
    Status:                True
    Type:                  PredictorReady
    Last Transition Time:  2022-01-10T03:33:23Z
    Severity:              Info
    Status:                True
    Type:                  PredictorRouteReady
    Last Transition Time:  2022-01-10T03:33:23Z
    Status:                True
    Type:                  Ready
  URL:                     http://resort.phantom.dev-predict-phantom.amh-group.com
Events:
  Type     Reason                    Age                   From                Message
  ----     ------                    ----                  ----                -------
  Warning  InferenceServiceNotReady  3m23s (x9 over 10d)   v1beta1Controllers  InferenceService [resort] is no longer Ready
  Warning  InternalError             3m18s (x27 over 10d)  v1beta1Controllers  fails to reconcile predictor: fails to update knative service: Operation cannot be fulfilled on services.serving.knative.dev "resort-predictor-default": the object has been modified; please apply your changes to the latest version and try again
  Normal   InferenceServiceReady     3m18s (x10 over 10d)  v1beta1Controllers  InferenceService [resort] is Ready

After about 5 minutes later, the version 0005 was terminated.

From the status, revision 5 was definitely not getting the traffic and the split is between version 6 and 7, wondering why version 5 was brought up for a brief time.

I am confused by it as well.  Especially when resources are insufficient, the old version brings up could result in pods which version are the latest can not start up.


Elliptic Curve code without ASM is broken with Clang 14

Hello,
It looks like on Linux (x86_64), starting with Clang 14, the code for Elliptic Curves (when assembler is not enabled) is broken when compiled with optimizations (O2/O3). It is totally fine when using assembler, but when using regular C code, some curves will fail in OpenSSL internals. It doesn't seem to be new (I tested a few older versions) and it seems to be also affecting the 3.0.x branch although I have personally experienced it on the 1.1.1 branch.
From running the tests, it seems to be only affecting a few curves (like secp384r1, prime192v1, prime256v1) and it seems to be that some validation does not pass, potentially due to bad codegen. I don't know if OpenSSL has some undefined behaviors in that section of the code as I haven't investigated yet. But, it was working well up to Clang 13, and GCC 11 is good too.
I've tested and generated logs for the following configs:

1.1.1n | Clang 14 | No ASM (broken)
1.1.1n | Clang 14 | ASM (working)
1.1.1n | GCC 11 | No ASM (working)

All these tests were done in a Docker container based on Ubuntu 22.04. I've also tested 1.1.1k | Clang 14 | No ASM and 3.0.2 | Clang 14 | No ASM which reproduce the exact same issue, so didn't generated logs for them.
config_1_1_1n_noasm_clang14.txt
tests_1_1_1n_noasm_clang14.txt
tests_verbose_1_1_1n_noasm_clang14.txt
config_1_1_1n_asm_clang14.txt
tests_1_1_1n_asm_clang14.txt
config_1_1_1n_noasm_gcc.txt
tests_1_1_1n_noasm_gcc.txt

This looks to be a problem with Clang optimising better in version 14.  Compiling with -fno-strict-aliasing seems to address the issue.

Thanks @paulidale for pointing out that it was not an issue when disabling strict aliasing. I've identified crypto/bn/bn_nist.c as being the file causing the issues. I'll see if I can identity the actual problematic code and update this ticket with my findings.

The issue seems to be around the function nist_cp_bn_0, which is called by all BN_nist_mod_XYZ functions except for BN_nist_mod_224 (likely why the 224 curve isn't affected). I don't see anything wrong with the nist_cp_bn_0 function, which is basically a manually written loop to memcpy + memset. It doesn't seem to violate strict aliasing rules from what I've seen as both the source and destination are BN_ULONG arrays.
I'm more a C++ programmer than a C programmer, so I might have missed something that is specific to C, but it can points you toward which section of the code is problematic. The rest of the BN_nist_mod_XYZ functions seem fine and the codegen doesn't seem to change that much except around the nist_cp_bn_0 call.
I've compared the generated code against when strict aliasing is disabled and it seems that the whole copy is discarded (and additional operations from the same else block?). So it might be a compiler bug. I've been able to make it generate seemingly proper assembly by rewriting the copy operation with an explicit call to memcpy and memset instead of the loop.
bn_nist_strict_aliasing.txt
bn_nist_no_strict_aliasing.txt
bn_nist_strict_aliasing_with_workaround.txt
I've dumped the preprocessed source file for easier debugging if it can help you:
bn_nist_preprocessed.txt
Just need to pass these additional arguments to the compiler to match the OpenSSL buildsystem: -fPIC -m64 -O3

Great work.  I agree that function should have no aliasing problems.
I've raised an issue with the LLVM folks.

I'm not sure that's right. I think there is a strict aliasing violation. C's strict aliasing and union rules are quite harsh. While there is a carve-out in C (but not C++, which is even stricter) for using unions to do type-punning, that only applies when you read or write directly to a union. Writing through a pointer to an inactive arm is UB.
See example 3 of <IP_ADDRESS> in n1256. (Whether these are the right rules is another matter. I doubt there is much C code out there that uses unions which is well-defined w.r.t. all this.)
Likewise, the strict aliasing rules (6.5, clause 7) doesn't include an allowance for unions in the effective type of the object, only the type of the lvalue expression (which doesn't involve a union).

Agreed, example 3 is damning.

This looks like the fundamental issue is if the array in the union is converted to a pointer or accessed directly as an array.  If it is converted, the behaviour is undefined.  The patch accesses the union as an array which solves this.
Interesting is that using memcpy/memset which both convert the array to a pointer, doesn't break the code.
At this point I'm satisfied that the problem is in the OpenSSL code.  I'm a little miffed that there were no warnings about undefined behaviour ever.

Are there plans to fix this for the 1.1.1 release?

Are there plans to fix this for the 1.1.1 release?

I created a backport or the fix in #18948


Flaky Test: Terminal - Terminal Tabs - should reset the tab name to the default value when no name is provided

See https://dev.azure.com/monacotools/Monaco/_build/results?buildId=278932&view=logs&j=4801dce2-64f3-53d6-b366-d49a1977c639&t=9ab437b3-ee70-5588-b120-442542e6d570

@meganrogge are you aware of any recent changes that could have affected this? It looks like the single tab is visible in the failure screenshot:

I think it's related to this line not clearing the quick pick:
https://github.com/microsoft/vscode/blob/be0e53510d8128ee4d17205fcb9511963a5bbb43/test/smoke/src/areas/terminal/terminal-tabs.test.ts#L45


Add rsqrt as a native triton language function

Adding support for rsqrt in Triton. Also adding tests for unary math ops.

OOC what does this lower to in PTX?
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-rsqrt is probably what people want, but it's "approx".

Yup, rsqrt.approx. We can add _rn, _rz versions of the function as we have done for other, if necessary.

Should we add a warning to the documentation that this is approximate and 1/sqrt(x) may be more accurate?  (Maybe the Triton function should even be called rsqrt_approx?)

We can add _rn, _rz versions of the function as we have done for other, if necessary.

I'm not sure these exist for rsqrt.approx?


Parallel circuit compilation

What is the expected enhancement?
In order to improve compilation times, parallelize circuit compilation would be a great addition.
I wanted to open this issue to discuss ideas and ways of implementing it in our current architecture.
The goal is to land an implementation for 0.6.

I made #701 fixes this issue and it seems the conversation is happening there.


V8: Express Forms - Files cannot be empty

If you have a form that has a file attribute attached to it, if you edit that entity without selecting a file, it errors out saying that you must specify a valid file. Requiring files should be optional.
And yes, the checkbox in options has required deselected

Checking in on this and I believe it to be resolved. There is an issue with attributes using the HTML file input attribute, but I think that can wait til 8.1


Deletion bug in utils.js

Hi,
Please read the edit below.
There is a bug which prevents deletion of records from the store:
createTombstone gets called with the user defined record: https://github.com/paldepind/synceddb/blob/master/client/src/utils.js#L116
But it has a hard coded scheme, which expects properties like key and version: https://github.com/paldepind/synceddb/blob/master/client/src/utils.js#L162
So in my stores, my primary key is id and not key, so when the tombstone gets put into the store, a new record is created (with a new id), which prevents the deletion from being noticed.
Possible solution: Pass the primary key field name and its value to createTombstone.
Thank you for your great work!
Edit: I just noticed that a property with name key seems to be generated automatically for newly created records. I'm using a db with existing entities, so it doesn't have that.

It is the expected behaviour for now, SyncedDB only supports key as a primary key. If we support arbitrary name for it, we have to identify the name for client and server, then we need a config file or something for that, which makes us confused. See also here.
But I agree with this can cause a migration problem like you for person who wants to use existing data.

Ah, I see. But isn't this rather unflexible?
Would you accept a PR which will still defaults to key but lets the user specify the key path, if desired? On the other hand, key seems to be a generated "unique" value, which probably has some other function I don't get yet... If key is essential for synceddb, maybe it could be a unique index instead, letting the user choose the key path. Just a few thoughts.
I'll read more into your code and IndexedDB the next few days before attempting any changes. :)

Actually it's @paldepind 's code 😃
Yeah let's do that, it should encourage adoption for such people. I think all we have to do is to make it able to pass keyPath option to both the client and the backend, then it should work.

I'll close this, since synceddb doesn't play well with my personal preferences for syncing. Some personal reasons/things that I noticed:

key is actually a generated unique id; it can't be just an auto incrementing number, because it could collide on the server side (if e. g. both a client A and a client B create different offline versions of an object with key 1).

Small bug: The MySQL persistence code creates a column key as INT, so the generated key doesn't get stored properly (using varchar should fix the problem).

So if I'd want a numeric, maybe even auto incrementing PK that is not identical to key, I'd have to heavily change the source code.

Making the property name key configurable would really mess up the source, because so many functions in utils.js use the key-property directly. A utils-Factory or sth. like that would be necessary in order the change the name.

Suggestion: Use property-prefixes like _sdb_ to indicate that a property doesn't belong to the "real" record.

Suggestion: Update the readme so ppl know they need to have a PK named key which must be computed by synceddb and can't be numerical.

Thanks for your support and respect for that sophisticated source code!

@whatda That list definitely helps us to think core arch, thanks. As you said offline app should care uniqueness of id in each records (any clustering DB env can also say the same thing in the first place) so auto increment is never be an option.
I still feel configurable keyPath is good for some people so let me keep open it. Also let me change the summary

@whatda That list definitely helps us to think core arch, thanks. As you said offline app should care uniqueness of id in each records (any clustering DB env can also say the same thing in the first place) so auto increment is never be an option.
I still feel configurable keyPath is good for some people so let me keep open it. Also let me change the summary


Add PDDS and 3 audio tasks to STAGING_PROJECT

Adds PDDS and 3 different audio tasks (for randomisation purposes) to STAGING_PROJECT
Range-info type for PDDS is not yet supported in current aRMT release so must release and update protocol at the same time
Task definitions are still in test branches

Should we add PDQ as well for testing


Update Polaris version

Breadcrumbs are broken in the version of Polaris being used, the issue was fixed in 1.7.0:
https://github.com/Shopify/polaris/issues/116

Closing because this project is now deprecated.


[Feature]: Support for webpack-assets-manifest

What problem does this feature solve?
Background
Shakapacker is a popular library for handling JS bundling in Rails apps. It's the successor to rails/webpacker and uses webpack-assets-manifest as a core component. There's ongoing discussion about adding Rspack as a bundler option for Shakapacker (see issue #476), but the lack of support for webpack-assets-manifest in Rspack is a significant roadblock.
Current Progress
I've forked the original webpack-assets-manifest plugin and started adapting it for Rspack. Most of the test coverage is passing, but there are still some issues. You can see the current state of the refactor here: rspack-assets-manifest test results.
To complete the port, Rspack needs to support or provide alternatives for the following webpack features:

codeGenerationResults: Used in the original plugin to gather asset information.
assetsInfo: Used to store and retrieve additional information about assets.

We could either:

Add similar properties or methods to Rspack's compilation object.
Provide alternative APIs that serve the same purpose but fit better with Rspack's architecture.
Think about workarounds or patterns to achieve the same results without these specific properties.

Benefits

Enable easier migration from webpack to Rspack for projects using Shakapacker.
Increase Rspack's compatibility with existing webpack ecosystems. (as the lib itself has >600k downloads/week)
Potentially benefit other projects that rely on webpack-assets-manifest.

I'm willing to contribute to this feature's implementation, but I would need some direction on Rspack's internal architecture and best practices for adding these or equivalent features.
What does the proposed API of configuration look like?
.

You can try to use rspack-manifest-plugin instead, it is forked from webpack-manifest-plugin and seems do the same thing with webpack-assets-manifest.
The assetsInfo can be got by compilation.getAsset()[].info. If you need to override it, you can use compilation.updateAsset(), but currently it does not support custom field (the rawRequest is not supported but the sourceFilename is supported).
Seems there is no method to get the code generation data of asset modules. But the assets of these modules will always be emitted through compilation.emitAsset and can be got from compilation.getAssets(). They have sourceFilename in their infos. You can also get all modules by compilation.modules so I think you can gather asset information by combining sourceFilename and module.userRequest/rawRequest.

Tried replace webpack-assets-manifest with rspack-manifest-plugin, but it doesn't support entrypoints as I see
according to this page rspack is compatible with webpack-assets-manifest, compilation fails because rspack.Compilation does not have codeGenerationResults property


gsutil cp -r two folder of same name cause error

if there is a A_folder in on B_bucket,
gsutil cp -r  A_folder B_bucket
will creat a A_folder within the original A_folder.

this happend during my mapping job.

likely job was interupted during the first gsutil cp -r sky_wordir ecker-rachel-analysis/mapping/IT-L23/mapping_000002.
Then the secod time there is sky_workdir within the folder.

ecker-rachel-analysis/m3c_merge_allc/c0_c8_c1_c1_r0 seems to have this issue, but the file is incomplete. Need more investigation


NPM modules ldap / ldapjs?

There are two packages on npm, ldap and ldapjs. The new version 1.0 is only published to ldapjs, not ldap. The npm documentation states:

Don't put "js" or "node" in the name. It's assumed that it's js, since you're writing a package.json file, and you can specify the engine using the "engines" field. (See below.)

So the ldap package should be the primary package, right?

This code is (and has always been) published as 'ldapjs' on npm.  The module named 'ldap' appears to be a clone, although it is now out of date.  If the owner of that name wishes to "donate" control to ldapjs, I'm happy to publish into both places.

@bluerival

If the owners of that module name grant me access, I'll publish there.

⚠ This issue has been locked due to age. If you have encountered a recent
problem that seems to be covered by this issue, please open a new issue.
Please include a minimal reproducible example
when opening a new issue.


muted_topics_ui: Rename mute_topic parameter to fix shadowing bug

This was broken by 6e6fb47e20ef690ddeae276de8c3d38f88d9073d (#24386).

Merged, thanks @andersk!
Probably medium term we will want to replace this API with something in the topic visibility policy framing; @prakhar1144 FYI.


Performance improvements obtained by caching model-specific data representations and resamplings thereof

Depends on: https://github.com/alan-turing-institute/MLJModelInterface.jl/pull/76 which introduces reformat and selectrows(::Model, I, args...) to the MLJ model API.
Goals
Addresses some issues raised in #309, and in particular:

Avoid the unnecessarily reconversion of data into a form specific to the model algorithm (eg, table -> matrix). Currently this happens every time model hyperparameter changes, or when training rows change.

Allow resampling of data to happen at level of model-specific representation. Currently resampling is applied to the user-representation (eg, table). This would also allow implementation to use matrix views, for example,  avoiding copying, if this is appropriate.

Avoid unecessary resamplings of the data. Currently data is resampled on every fit! after a model hyperparameter change, even if the training rows are unchanged.

This PR is a proof of concept achieving these goals by:

Adding new fields data and resampled_data to machines to cache model-specific representations of the data, and resampled versions respectively

Change the logic in fit_only! to cache these representations, making use of the two methods reformat and  selectrows(::Model) in such a way that the fallbacks given in MLJModelInterface result in essentially the old behaviour.

Main features proposed solution

Buying into the improvements is optional for existing (or new) implementations of the extended model API and so can be phased in gradually.

Changes suggested here are only breaking to users who access current "low-level" methods fit and update, and only for those models that choose to buy into the performance improvements by implementing reformat, selectrows(::Model) and altering their fit/predict (see "Buying into the extended API" below). Users who interact using fit! should not encounter breaking behaviour.

Tested locally with the MLJModelInterface PR

The current PR is limited to achieving the goals above on the training side, not on the predict/transform side

Buying into the extended API
Here's an example of how to buy into the the new API, which is tested in detail here
Existing implementation
function MLJModelInterface.fit(model::SomeModel,
                               verbosity,
                               X,
                               y)

    A = MLJModelInterface.matrix(X)
    fitresult = ...

    return fitresult, cache, report
end

function MLJModelInterface.predict(::SomeModel, fitresult, Xnew)
    Anew = MLJModelInterface.matrix(Xnew)
    return Anew*fitresult
end

Additions and changes
The signature of fit is changed to accept matrix A in place of table X, and the conversion to matrix is removed, with that role being played by the overloaded reformat method:

function MLJModelInterface.fit(model::SomeModel,
                               verbosity,
                               A,
                               y)
    fitresult = ...

    return fitresult, cache, report
end

MLJModelInterface.reformat(model::SomeModel, X, y) = (MLJModelInterface.matrix(X), y)

If update is also overloaded, then a similar change is made there.
Finally, we must specify how to resample the model-specific representation of the data:
MLJModelInterface.selectrows(model::SomeModel, I, A, y) = (view(A, I, :), view(y, I))

That's it.

Thoughts anyone?
@OkonSamuel @tlienart @yalwan-iqvia @jeremiedb

So cursory glance this seems good to me ( I didn't touch the code changes yet )
One of my main remarks would be that this seems geared towards implementors buying into the performance enhancements, however, I think a level of control needs to be afforded to the user on a case by case basis as to whether the implementation should buy in -- why?
Well for some modelling/system setups, modelling might be getting near to memory limits in system. In this case caching resamplings can make the difference between running/not running.
In this case, I'd imagine user wishes to opt out in order to have it running with degraded performance, vs not running at all.
Does current implementation provide any mechanism for this?

P.s. Thanks for this
In LightGBM the system current has to construct LGBM specific data-sets for each resampling of data prior to fitting. This can actually be computationally non-trivial. By providing a mechanism by which we can hold ontoi our own form of the data and then producing our own resamples, this cost can almost entirely be negated. In fact, in my tests, the subsampling of the specific data structure is much faster than even just vanilla matrix slicing for large data sets (multi GBs +)

@yalwan-iqvia Thanks for the feedback!

In this case, I'd imagine user wishes to opt out in order to have it running with degraded performance, vs not running at all.

Maybe worth mentioning here that there are two kinds of caching - caching the model-specific representation of the whole training data set (before resampling) - and caching a particular subsample. Whether each adds to memory requirements depends both on the model implementation and what form the user-data takes. For example, if the "table" presented by the user is just a wrapped matrix (a MatrixTable) and the model algorithm is just unwrapping it, then there is no extra memory cost to "caching" the data. We are just pointing to stuff inside the user-specified data. Similarly, if subsampling of a matrix is just constructing a view, then again no extra memory cost.
But, yes an "optimize_memory" option makes sense and can be added. However a particular model may want to also provide options - such as whether to construct a view of data or make a copy when restricting to specified rows of a matrix.
Can one construct a "view" of the model-specific representation in LightGBM (DMatrices?) (a view that can be consumed by fit). Or does indexing necessarily create copies?
Be great if you could have a stab at implementing the extended API for LightGBM in a fork and do some benchmarking. The two branches you need - from MLJModelInterface - and from MLJBase for testing - are both called "mlj2".
FYI I am on leave for the next week.

Can one construct a "view" of the model-specific representation in LightGBM (DMatrices?) (a view that can be consumed by fit). Or does indexing necessarily create copies?

I think so
Basically you can create a Dataset object and then on top of this there is a getsubset API which returns a new dataset reference, but when I tested this with whomper data and then obtained a subset dataset I didnt see the memory usage change -- I believe (without looking at the C++ source code) that it just just a new dataset object which references the old one and additionally stores the subset elements request for use during training steps. Since this is still a dataset object it is consumable by lightgbm for subsequent fit calls.
We actually put a PR on the mainline code to allow this specialisation and performance improvement just recently:
https://github.com/IQVIA-ML/LightGBM.jl/pull/80/files
Idea was rather than requiring matrices, refactor out the dataset construction element and allow user to provide own dataset(s). This would then make it straightforward for us to support the paradigm within the MLJ wrapper (as the "user").
I can't promise I can get around to trying out the extended API over the next week but I'll certainly see what I can do

Sorry being late to the show. Being able to circumvent the overhead was definitely a concern on my side, very nice to see an answer to it! I'll be testing the proposed buy-in from the mlj2 branches with EvoTrees and hopefully there won't be any noticeable difference with the native API. I'll get back to you shortly. Let me know if there were any other aspect you'd like me to check.

So the outcomes look positive from using the mlj2 branches.
For the model setup and initial iterations, the timing remains similar, but with a good reduction in the memory footprint:
Old:
julia> @time fit!(tree, rows=train, verbosity=1, force=false)
[ Info: Training Machine{EvoTreeRegressor{Float32,…}} @230.
  3.089232 seconds (22.55 k allocations: 2.899 GiB, 18.94% gc time)

MLJ2:
julia> @time fit!(tree, rows=train, verbosity=1, force=false)
[ Info: Training Machine{EvoTreeRegressor{Float32,…}} @882.
  2.823589 seconds (21.94 k allocations: 1.701 GiB, 4.42% gc time)

Where things are interesting, is that while the backen update function was already performing well, the issue was coming from the call to fit! it situations of simple model update. On the new mlj2 branches, that fit! function now performs pretty as fast as the underlying update.
Old:
tree.model.nrounds += 1
julia> @time update(tree.model, 0, tree.fitresult, tree.cache, X, Y);
  0.068055 seconds (1.94 k allocations: 41.474 MiB)

julia> @time fit!(tree, rows=train, verbosity=1)
[ Info: Updating Machine{EvoTreeRegressor{Float32,…}} @230.
  0.684355 seconds (2.78 k allocations: 658.003 MiB, 4.51% gc time)

New:
tree.model.nrounds += 1
julia> @time update(tree.model, 0, tree.fitresult, tree.cache, X, Y);
  0.074739 seconds (1.95 k allocations: 41.473 MiB)

julia> @time fit!(tree, rows=train, verbosity=1)
[ Info: Updating Machine{EvoTreeRegressor{Float32,…}} @882.
  0.085758 seconds (2.22 k allocations: 41.507 MiB)

Which is good to see!
Some small notes about implementation.
To restrain the dependency to MLJModelInterface, the MLJBase.matrix is changed for:
MLJModelInterface.reformat(model::EvoTypes, X, y) = (MLJModelInterface.matrix(X), y)

For the metadata_model, I've comment out input=MLJModelInterface.Table(MLJModelInterface.Continuous).  What would be the proper way to specify the union of Table and Matrix?
Finally, this is a bit tangent, but has there been any adopted design for supporting the particular iterative training nature of boosted models? I'm referring to the difficulty from my perspective from using MLJ for such models given that the number of iterations isn't treated as other hyper-parameters, but rather more as an upper bound while the model is trained till an early stop criteria is met. It is over such sequence of iterations that one would want to perform an hyper search for GBT, but it doesn't appear trivial to integrate a tracking metric of an eval dataset in MLJ for that purpose, maybe I've overlooked this? That early stop over an eval data is something commonly embedded in GBT libraries, whether it's xgboost, lightgbm or evotrees.

@jeremiedb Thanks for that very helpful and encouraging feedback!

Some small notes about implementation.
To restrain the dependency to MLJModelInterface, the MLJBase.matrix is changed for:

Good catch, thanks. Updated my original post above.

For the metadata_model, I've comment out input=MLJModelInterface.Table(MLJModelInterface.Continuous). What would be the proper way to specify the union of Table and Matrix?

Are you saying you needed to comment this out to get the mlj2 branches to work? If so, please post the error.
Or you want to allow both forms of input? Yes; you can do this:
input=Union{MLJModelInterface.Table(MLJModelInterface.Continuous), AbstractMatrix{MLJModelInterface.Continuous)}

But this means the X received by fit in the old MLJ could be either type (and you have to process appropriately) or, in mlj2, you have two reformat methods. For example, something like:
_reformat(::Type{<:Table}, X) = MLJModelInterface.matrix(X)
_reformat(::Type{<:AbstractMatrix}, X) = X
MLJModelInterface.reformat(model::SomeModel, X, y) = (_reformat(scitype(X), X), y)

Finally, this is a bit tangent, but has there been any adopted design for supporting the particular iterative training nature of boosted models? I'm referring to the difficulty from my perspective from using MLJ for such models given that the number of iterations isn't treated as other hyper-parameters, but rather more as an upper bound while the model is trained till an early stop criteria is met. It is over such sequence of iterations that one would want to perform an hyper search for GBT, but it doesn't appear trivial to integrate a tracking metric of an eval dataset in MLJ for that purpose, maybe I've overlooked this? That early stop over an eval data is something commonly embedded in GBT libraries, whether it's xgboost, lightgbm or evotrees.

Yes, this is a very high priority (https://github.com/alan-turing-institute/MLJ.jl/issues/139). Appreciate your patience.

Are you saying you needed to comment this out to get the mlj2 branches to work? If so, please post the error.

No, it was just a confusion on my side where I assumed MLJ2 would be expected to directly support support Matrix. Actually, the Union for Tables and Matrix is right what I wanted, so both Tables and Matrix goes smoothly, even without adding the double _reformat trick.
No worry about the evaluation metric tracking, it's just that I recently had some reflections on whether it whether we should consider transfer our ML pipeline from R to Julia, and to what extent MLJ could be piggy-backed.
Do you have any rough timeline as to when you expect to have those MLJ2 released?

@jeremiedb I've now a pretty good idea how the improvements can be carried over to the prediction side.  After Christmas, I'll add this and we should be good to go. So, hopefully early January for mlj2.

Update:
I have now added:

Option for a machine to turn off the data caching. This is implemented with a new type parameter C for machines, now instances of Machine{M<:Model, C} where C is (true or false). To instantiate a machine with no caching user does machine(model, data...; cache=false).

Data caching on the predict side (and for any other operation, except inverse_transform). This requires the implementer to: (i)  further overload reformat(model, args...) and selectrows(model, I, data...) for the case that length(args)=1 and length(data)=1 (unless model is Unsupervised, where this would have been done already on the training side); and (ii) refactor every operation (such as predict, but except for inverse_transform) to expect its data arguments to be reformated versions of their original ones.

Details are given in a proposed documentation PR at MLJ.

Update:
I have now added:

Option for a machine to turn off the data caching. This is implemented with a new type parameter C for machines, now instances of Machine{M<:Model, C} where C is (true or false). To instantiate a machine with no caching user does machine(model, data...; cache=false).

Data caching on the predict side (and for any other operation, except inverse_transform). This requires the implementer to: (i)  further overload reformat(model, args...) and selectrows(model, I, data...) for the case that length(args)=1 and length(data)=1 (unless model is Unsupervised, where this would have been done already on the training side); and (ii) refactor every operation (such as predict, but except for inverse_transform) to expect its data arguments to be reformated versions of their original ones.

Details are given in a proposed documentation PR at MLJ.

Update:
I have now added:

Option for a machine to turn off the data caching. This is implemented with a new type parameter C for machines, now instances of Machine{M<:Model, C} where C is (true or false). To instantiate a machine with no caching user does machine(model, data...; cache=false).

Interface point in Resample objects for specifying what cache parameter to use in machines constructed in resampling. This is tested in the following MLJTuning PR, which passes locally.

Data caching on the predict side (and for any other operation, except inverse_transform). This requires the implementer to: (i)  further overload reformat(model, args...) and selectrows(model, I, data...) for the case that length(args)=1 and length(data)=1 (unless model is Unsupervised, where this would have been done already on the training side); and (ii) refactor every operation (such as predict, but except for inverse_transform) to expect its data arguments to be reformated versions of their original ones. This only avoids an extra reformat in the case of a call like predict(mach, rows=...), and resampling has been refactored to call predict(mach, rows=...) instead of predict(mach, Xtrain) to make use of this.

Details are given in a proposed documentation PR at MLJ.

Update:
I have now added:

Option for a machine to turn off the data caching. This is implemented with a new type parameter C for machines, now instances of Machine{M<:Model, C} where C is (true or false). To instantiate a machine with no caching user does machine(model, data...; cache=false).

Interface point in Resample objects for specifying what cache parameter to use in machines constructed in resampling. This is tested in the following MLJTuning PR, which passes locally.

Data caching on the predict side (and for any other operation, except inverse_transform). This requires the implementer to: (i)  further overload reformat(model, args...) and selectrows(model, I, data...) for the case that length(args)=1 and length(data)=1 (unless model is Unsupervised, where this would have been done already on the training side); and (ii) refactor every operation (such as predict, but except for inverse_transform) to expect its data arguments to be reformated versions of their original ones. This only avoids an extra reformat in the case of a call like predict(mach, rows=...), and resampling has been refactored to call predict(mach, rows=...) instead of predict(mach, Xtrain) to make use of this.

Details are given in a proposed documentation PR at MLJ.

Re-opening as error appears to be OpenML connection related.

Re-opening as error appears to be OpenML connection related.

closing in favour of #501

closing in favour of #501