POC separate queue

While trying to optimize the queue, I realized most of the data in Job collection is unnecessary to get the next waiting job.
See from a total of 8562 records:

Status
Count
%

waiting
2835
33.11%

success
4614
53.89%

error
840
9.81%

started
24
0.28%

cancelled
249
2.91%

This means that the get_next_waiting function would only need about 34% of the data in the collection.
We have seen some months ago (and whenever we have a high load in the queue) that MongoDB complains about Scanned Objects / Returned has gone above 1000.
I propose to split the queue data into two different collections:
One for the active jobs (started and waiting) and another for the finished ones (success, error, and canceled).
This approach would help to optimize the query duration and also would target the index definition in a better way.
There is a lot to do yet regarding this PR:

Evaluate if it is needed MongoDB migrations for the new collection
Fix tests
Fix style
Modify indexes based on requirements

Before proceeding, let me know wdyt @huggingface/datasets-server

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (f41c757) 90.50% compared to head (062924d) 85.01%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2137      +/-   ##
==========================================
- Coverage   90.50%   85.01%   -5.50%
==========================================
  Files         250       33     -217
  Lines       15441     1648   -13793
==========================================
- Hits        13975     1401   -12574
+ Misses       1466      247    -1219

Flag
Coverage Δ

jobs_cache_maintenance
?

jobs_mongodb_migration
?

libs_libcommon
?

services_admin
?

services_api
83.44% <ø> (ø)

services_rows
82.72% <ø> (ø)

services_search
80.48% <ø> (ø)

services_sse-api
94.21% <ø> (ø)

services_worker
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

The finished jobs are kept only to be able to report metrics, and are deleted after 10 minutes. If we're able to change the way we collect metrics, we could even get rid of them completely

The finished jobs are kept only to be able to report metrics, and are deleted after 10 minutes. If we're able to change the way we collect metrics, we could even get rid of them completely

Yes, it is only used in count_finished_jobs, which is needed for the cron job to collect queue metrics.
This job helped us do a cleaning/regularization of the metrics, but the value of the metric is even not totally confident since the table has a TTL, I would say we can safely stop storing the finished jobs, or as I proposed in this PR, create a new collection (Maybe it can have even less attributes like unicity_id, job_type and status) and compute metrics from there.

as the workers can now generate metrics, I would say we should delete the jobs and register a new metric on every finished job

As per our last conversation with @severo, I will close this PR in favor of another one that will remove the finished jobs and then we will store the metrics for them in a different Prometheus Counter (Instead of our incremental metrics).


Need Assistance: Troubleshooting "OCI runtime create failed" error when deploying 'e2e-test-images/agnhost' via "Hello Minikube" tutorial on Windows

11h         Warning   Failed                    pod/hello-node-7c7b95ff98-phftk    Error: failed to start container "agnhost": Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "C:/Program Files/Git/agnhost": stat C:/Program Files/Git/agnhost: no such file or directory: unknown
I get this error, when I try to create a deployment (link for instructions).
Steps to reproduce:

minikube start --driver=docker
in other terminal minikube dashboard --url
in the first terminal kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- /agnhost netexec --http-port=8080 (regarding instructions)

My OS: Windows 10 home. I use docker driver, because Hyper-V doesn't exist in windows 10 home.
Could you explain please, how to fix the issue.

Page mentioned in issue (based on the issue title): https://kubernetes.io/docs/tutorials/hello-minikube/
/language en

/retitle Need Assistance: Troubleshooting "OCI runtime create failed" error when deploying 'e2e-test-images/agnhost' via "Hello Minikube" tutorial on Windows

Looks like a support request in its current form.
/kind support

I could create deployment.
I used the following command:
kubectl create deployment hello-minikube --image=kicbase/echo-server:1.0 (from section 4).
Maybe the problem is related to image which is described in the tutorial?

The issue seems to occur when you're passing commands and arguments to the kubectl create deployment command on Windows.
kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- /agnhost netexec --http-port=8080
Could you retrieve the manifest of your deployment by running the following command?
kubectl get deployments.apps hello-node -o yaml
Please check what is present under this section (I have pasted the output I receive on MacOS)
    spec:
      containers:
      - command:
        - /agnhost
        - netexec
        - --http-port=8080
        image: registry.k8s.io/e2e-test-images/agnhost:2.39

Please check if it is adding the C:/Program Files/Git/ in the command for the container.

Yes, the line C:/Program Files/Git/agnhost exists in command section.
- command:
      - C:/Program Files/Git/agnhost
      - netexec
      - --http-port=8080
      image: registry.k8s.io/e2e-test-images/agnhost:2.39
      imagePullPolicy: IfNotPresent

in status section I have the following:
`status:
conditions:

lastTransitionTime: "2024-05-08T09:02:08Z"
lastUpdateTime: "2024-05-08T09:02:08Z"
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: "False"
type: Available
lastTransitionTime: "2024-05-08T09:02:08Z"
lastUpdateTime: "2024-05-08T09:02:08Z"
message: ReplicaSet "hello-node-fb6d66547" is progressing.
reason: ReplicaSetUpdated
status: "True"
type: Progressing
observedGeneration: 1
replicas: 1
unavailableReplicas: 1
updatedReplicas: 1
`

@SergeiPavlitz
I am thinking that this could be a bug in the kubectl client library for Windows.
The deployment is created using an imperative command through kubectl along with passing the commands to the container from the command line. The actual manifest which is created unnecessarily concatenates some Windows specific path (C:/Program Files/Git) which obviously will not work on the container.
I would suggest to raise an issue to kubectl to get an opinion on it.
https://github.com/kubernetes/kubectl/issues

Thank you. A new issue is created.

The new issue it created

I have also encountered the same problem.
Is there a way to get it to work using 'apply' and updating the yaml file and saving it locally:
kubectl apply -f deployment.yaml
What would be the location of agnost?
What can we replace this line with to get it to work?
C:/Program Files/Git/agnhost
   ...
    spec:
     containers:
     - command:
       - C:/Program Files/Git/agnhost
       - netexec
       - --http-port=8080

Having the same error on windows:
spec:
containers:
- command:
- C:/Program Files/Git/agnhost
- netexec
- --http-port=8080
image: registry.k8s.io/e2e-test-images/agnhost:2.39
imagePullPolicy: IfNotPresent
name: agnhost


Switch to new CouchbaseClient API

From<EMAIL_ADDRESS>on March 05, 2012 20:00:26
Cpuchbase Server 1.8 (formerly known as Membase) is using a slightly different API and it's no longer part of the spymemcached API.
Original issue: http://code.google.com/p/memcached-session-manager/issues/detail?id=126

From martin.grotzke on March 05, 2012 11:04:19
(No comment was entered for this change.)
Status: Accepted
Labels: -Type-Defect Type-Enhancement

From martin.grotzke on July 27, 2012 01:52:53
Related:  issue #149

From<EMAIL_ADDRESS>on March 16, 2013 13:45:35
see my pull request for the switch to the new API: https://github.com/magro/memcached-session-manager/pull/23

From martin.grotzke on March 28, 2013 02:13:38
 Issue 149  has been merged into this issue.

From martin.grotzke on March 28, 2013 02:14:26
Thanx for the pull request! It's merged and released.
Status: Fixed
Labels: Milestone-1.6.4


NIFI-9786 Add debug logs if keystore cannot be loaded

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables logging for KeyStoreUtils isStoreValid validation failure; closed improvement NIFI-9786.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[-] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[x] Have you verified that the full build is successful on JDK 11?
[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Sample exception:
2022-03-21 16:49:41,127 DEBUG [Validate Components Thread-2] o.a.nifi.security.util.KeyStoreUtils Keystore [file:/Users/tamashorvath/repos/cloudera-truststore.jks] Type [JKS] load failed java.io.IOException: Keystore was tampered with, or password was incorrect 	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:782) 	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) 	at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) 	at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) 	at java.security.KeyStore.load(KeyStore.java:1445) 	at org.apache.nifi.security.util.KeyStoreUtils.isStoreValid(KeyStoreUtils.java:399) 	at org.apache.nifi.ssl.StandardSSLContextService.validateTruststoreFile(StandardSSLContextService.java:495) 	at org.apache.nifi.ssl.StandardSSLContextService.validateTruststore(StandardSSLContextService.java:459) 	at org.apache.nifi.ssl.StandardSSLContextService.validateStore(StandardSSLContextService.java:376) 	at org.apache.nifi.ssl.StandardSSLContextService.customValidate(StandardSSLContextService.java:197) 	at org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:130) 	at org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:741) 	at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:708) 	at org.apache.nifi.controller.service.StandardControllerServiceNode.performValidation(StandardControllerServiceNode.java:500) 	at org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:722) 	at org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52) 	at org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:42) 	at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 	at java.lang.Thread.run(Thread.java:748) Caused by: java.security.UnrecoverableKeyException: Password verification failed 	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) 	... 24 common frames omitted


Docker error while installing Orchest: too many levels of symbolic links

I am getting the below error while installing orchest using 'bash orchest install':
docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/a3534df899b9d4314c714596bab8b515cedc0af829f59785d5e701b4dada794e-init/merged: too many levels of symbolic links
I am following the steps mentioned here: https://docs.orchest.io/en/stable/getting_started/installation.html using WSL2 on windows 10.

Hi @megha120. Sorry to hear that the installation is not working for you.
I've not seen that error before, nor am I running on WSL2 myself so I will just make an educated guess on what causes the issue. Looking at Docker related issues (e.g. https://github.com/docker/for-win/issues/5763) it seems that the issue could be related due to available disk size.
In order to free up disk space, you could run docker system prune and then try the install again.
Hope that works! Let me know if it doesn't :innocent:

Did freeing up disk space solve the issue for you? Would love to better understand where thinks are breaking for you to improve our installation documentation (and of course so that you can enjoy using Orchest).

I am considering the issue to be stale, please feel free to reopen.


Styleing completionwindow

After trying to set the style of the CustomCompletionWindow i found out this cant be done because it is a private class.
Describe the solution you'd like
Convert the private CustomCompletionWindow class to a public class so it can be styled.
Describe alternatives you've considered
A property can be added to set the style of the completion window.

CustomCompletionWindow inherits from CompletionWindow and doesn't change the style key, so you could use the base class to style it.


Unable to prepare setup for platform keyatome.sensor

Intregration is not working
To Reproduce
Steps to reproduce the behavior:
Reboot after installing keyatome integration using HACS
Logs
Logger: homeassistant.setup
Source: setup.py:317
First occurred: 14:27:26 (1 occurrences)
Last logged: 14:27:26
Unable to prepare setup for platform keyatome.sensor: Platform not found (cannot import name 'SensorDeviceClass' from 'homeassistant.components.sensor' (/volume2/@appstore/homeassistant/env/lib/python3.8/site-packages/homeassistant/components/sensor/init.py)).

Hello
Thanks for using the components !
Please can you tell me which version of HA you are using ?

Hi Jugla,
I am using Home Assistant 2021.9.7.
It is a Home assistant core version, the latest one available for synology NAS.

Unfortunately i dont think it is possible on Synology system, but i will dig into it.
Thank you for your support. :)

Thanks !
I will close the the issue.
Let me know if you succeed in installing the new HA release


[Feature request] Local Authentication

Hello,
Thanks for making the app. I just realize I can only login via oauth. I really need the feature to authenticate without 3rd party though... Please implement it in the future.

You can achieve local login through the admin interface.
First, access the shell of the tabby-web container:
docker exec -i -t tabby-web /bin/sh

Create a superuser:
/ # ./manage.sh createsuperuser
Username: admin
Email address: ***@***
Password:
Password (again):
Superuser created successfully.

Once created, navigate to https://your-domain.com/admin to log in. After a successful login, you’ll be redirected to the admin page. You can manually return to the homepage at https://your-domain.com.


Help component : Uncaught TypeError: Cannot read property 'contains' of undefined

Issue and Steps to Reproduce
Uncaught TypeError: Cannot read property 'contains' of undefined at HTMLDocument.handleEvent when clicking on the Help component or an input next to it.
Versions
2.1.15
Screenshots
Expected
Actual
Additional Details
Can be reproduced in the storybook :
https://axaguildev.github.io/react-toolkit/latest/storybook/index.html?path=/story/form-input-text--textinput

Fixed in previous PR


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_41a63nzme8/clone/recipe/meta.yaml:15: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_41a63nzme8/clone/recipe/meta.yaml:20: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_41a63nzme8/clone/recipe/meta.yaml:29: missing_pip_check: For pypi packages, pip check should be present in the test commands
WARNING: /tmp/abs_41a63nzme8/clone/recipe/meta.yaml:38: missing_description: The recipe is missing a description
Errors were found


Function to email folks who are missing annotations

@jaybee84 @kthrog - can you help me scope this function? What would you want this to do? (This is a follow up on the conversation we had with JOB in our last biweekly).

So, this is just a rough idea, but here's a sketch of what I'm thinking, and how it might be executed.
What I'd like:

to receive an email when a project has gone a certain amount of time (e.g., 6 months, or maybe 1 year is more realistic) since creation, with no new files added
to receive an email when a project's files have gone a certain amount of time (e.g., 1 month) with no annotations (or at least blank values for keys we deem essential)
in a perfect world, I'd like those emails to then allow me to trigger an templated email to the investigator saying either a) we've noticed you haven't uploaded any files, please see docs here, contact us if needed or b) we've noticed you haven't added annotations to the files you've uploaded, please use the data curator app to annotate, contact us if needed

How it might be done:

using a couple of keys we deem essential/should never be blank in annotations considered complete (e.g., resourceType, dataType, assay), look for blank values in those keys in the Portal - Files table and compare with file creation date -- if more than the allotted period of 'annotation time' (e.g., 1 month) has passed, send an email
if we wanted to get really fancy, perhaps we could do the above, and then merge with data from the IncomingData portal table, add a column to start adding the end of award date (we don't currently collect this, but could, for all DSPs that completed it), and if the end of award date < 3 months away, notify everyone (us, funder, investigator) and send a templated email with instructions

Will keep thinking about this!

Closing as scoped and implemented in https://github.com/nf-osi/jobs/issues/23


Make it possible to OR slot values

Description of Problem:
Currently only one slot value can be given  per slot_was_set step or condition. It would be useful to be able to "OR" slot values, i.e. list multiple options that all qualify for that step, especially with many-valued categorical slots.
Currently the only way around is either to create "meta slots" which reflect a certain grouping of slot values, or to create individual stories for each slot value, even when the steps in each one are identical otherwise. This can lead to an explosion in the number of stories with a many-valued categorical slot.
Overview of the Solution:
Allow listing multiple allowable values per slot in a story or rule slot_was_set step (see examples). If in a conversation the slot is set to any one of those values, the dialogue state should be considered to match the story/rule at that point.
Examples (if relevant):
For example, given this slot:
slots:
  account_type:
    type: categorical
    values:
       - free
       - standard
       - gold
       - premium

There could be cases where you want to group the values into free, and standard, gold, premium,
and other cases where each individual value gets its own variant on a story, or the values are grouped differently:
stories:
   story: greet a customer on a free plan
   steps:
      - intent: greet
      - slot_was_set:
         - account_type: free
      - action: utter_greet_free_customer

   story: greet a customer on any paid plan
   steps:
      - intent: greet
      - slot_was_set:
         - account_type:
               - standard
               - gold
               - premium
      - action: utter_greet_paying_customer

   story: list options standard and free
   steps:
      - intent: what_can_do
      - slot_was_set:
         - account_type:
               - standard
               - free
      - action: utter_standard_options

   story: list options gold and premium
   steps:
      - intent: what_can_do
      - slot_was_set:
         - account_type:
               - gold
               - premium
      - action: utter_advanced_options

Blockers (if relevant):
None known
Definition of Done:

[ ] Feature added
[ ] Tests are added
[ ] Feature described the docs
[ ] Feature mentioned in the changlog

@melindaloubser1 is this issue open for community? I would like to contribute.

@melindaloubser1 is this issue open for community? I would like to contribute.

It's still under discussion, so I'm not sure (@TyDunn )

It's still under discussion, so I'm not sure (@TyDunn )

Hi @kaustuk. If you are interested in contributing this, could please sketch out a plan for how you would go about doing it?

Hi @kaustuk. If you are interested in contributing this, could please sketch out a plan for how you would go about doing it?

@TyDunn thanks for the reply. Sure I will come up with plan by this weekends.

@TyDunn thanks for the reply. Sure I will come up with plan by this weekends.

@kaustuk Thanks! This will help us figure out if we feel this can be handled by the community 🚀

@kaustuk Thanks! This will help us figure out if we feel this can be handled by the community 🚀

at least four customers (including one of the biggest) have asked for this now so bumping it up in CSE priority

closing in favour of #8933


Revert "[Ubuntu] install gcc@11 from brew (#6202)"

This reverts commit d2c84c9d4c30a469ab32e88d9118914a9380a53f.

Every single linux binary relies on glibc from brew now, in the screenshot ldd /bin/ls
Related issue:
Check list

[ ] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

Hi, I've noticed failing github action runs caused by playwright missing packages for and libexpat.so.1 and libgcc_s.so.1 on ubuntu-20.04 20220912.1 at the time marked as pre-release and rolled out 20%.
Could you help me out and confirm: Was this issue the reason and is now being rolled back? Thanks

@andys8 you are likely to hit this exact problem, we are rolling out changes back for the time being

Thanks for confirming 👍

I am also getting exactly the same playwright error as @andys8 mentioned. The issue is breaking our e2e test pipeline and blocking our code check-ins.
@mikhailkoliada is there a way to add some test cases to validate the image before it is published?

The issue is that you're picking up Homebrew ldd when doing ldd /bin/ls. See workflow run here: https://github.com/carlocab/workflow-test/actions/runs/3053860442/attempts/1#summary-8353478779
ldd probably shouldn't be in PATH. A workaround in the meantime is to brew unlink glibc. This may break some Homebrew-installed things. But zstd should be fine for now.

@mikhailkoliada we are using the larger GitHub runners and unfortunately, Ubuntu 20 already has the buggy version 20220912.1. Is there any way to update it?

@mikhailkoliada we are using the larger GitHub runners and unfortunately, Ubuntu 20 still has the buggy version 20220912.1. Is there any way to update it to 20220905.1?

Try calling brew unlink glibc in your workflow.

@mcarrillowz Could you please provide more details? Where do you see the old image version and how?

@mikhailkoliada we are using the larger GitHub runners and unfortunately, Ubuntu 20 still has the buggy version 20220912.1. Is there any way to update it to 20220905.1?

Try calling brew unlink glibc in your workflow.

Unfortunately, I'm running the job within a container and can't unlink the library before starting it 😞

@mcarrillowz Could you please provide more details? Where do you see the old image version and how?

When I create a runner using Ubuntu 20:

@mcarrillowz large  runners will be rolled back in an hour or so, sorry for getting you in troubles!

@mcarrillowz large runners will be rolled back in an hour or so, sorry for getting you in troubles!

Great, it would be great if you keep me posted, please 👍


README.md

/cc @kyrtapz @flavio-fernandes
/assign @squeed

Gave it a quick pass, seems reasonable.

/approve
/lgtm
let's get this in; any additional changes can be in another PR


Status of unused dependencies

There are a few unused dependencies

openssl_sys
ring

ring is supposed to solve build issue on Apple M1 architecture. But is it still needed?
Someone mentioned that exa had the issue and ring can solve it. But how? There's no
use ring in the entire crate. Also, I don't see anything mentioned about ring, either
in its manifest, or in the homebrew recipe: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/exa.rb
Also, ring at least should be gated behind a cfg attribute for darwin only.
Similarly, openssl_sys is never used. Could that be safely removed?

hubcaps depends on jsonwebtoken 7, which depends on ring, which is not ready for Apple Silicon. The advice is to explicitly depend on a newer version which does. (See cargo tree --invert ring in lychee.)
openssl_sys can't build on macOS with the openssl bindings (issue), which is why we use ring to statically link SSL support.
At least that's what I think is going on. It's all a bit of a mess. 🤷♀

But I'd be more than happy to remove that or at least put it behind a gate.
Also, maybe we should be good OSS citizens and get send PRs to the deps in question.

Also I think we should have a new feature flag serde that controls whether serde is enabled for dependencies.

Good idea. 👍

As for ring and openssl_sys, I confirm that they cannot be built for aarch-apple-darwin target.
hubcaps requires ring. So one simple solution is to replace it with github_rs.

Can we close this?

not really..
They're not added to optional dependencies yet.

This has been in the queue for a while. @lebensterben do you want to tackle this? I personally don't have the resources right now. 😕

Closing this to keep the issue tracker clean. @lebensterben, feel free to reopen if you're planning to work on this. 😉


Adds failing spec and fix for chaining of OR criteria

Currently, if you call .scroll on a mongoid criteria with an $or clause, the criteria.where(...) line does not separate the $or conditionals from the cursor criteria and the original criteria.
Example:
=> criteria
=> #<Mongoid::Criteria
  selector: {"$or"=>[{"a_time"=>{"$gt"=>2015-07-22 05:02:03 UTC}}, {"a_time"=>{"$lte"=>2015-07-22 05:02:03 UTC}, "a_date"=>{"$gte"=>2015-07-30 00:00:00 UTC}}]}
  options:  {:sort=>{"a_time"=>1}, :limit=>2}
  class:    Feed::Item
  embedded: false>

=> cursor.criteria
=> {"$or"=>[{"a_time"=>{"$gt"=>2015-02-22 06:02:03 UTC}}, {"a_time"=>2015-02-22 06:02:03 UTC, :_id=>{"$gt"=>BSON::ObjectId('55b0088c5361725886000001')}}]}
```

```
=> criteria.where(cursor.criteria)
=> #<Mongoid::Criteria
  selector: {"$or"=>[{"a_time"=>{"$gt"=>2015-07-22 05:02:03 UTC}}, {"a_time"=>{"$lte"=>2015-07-22 05:02:03 UTC}, "a_date"=>{"$gte"=>2015-07-30 00:00:00 UTC}}, {"a_time"=>{"$gt"=>2015-02-22 06:02:03 UTC}}, {"a_time"=>2015-02-22 06:02:03 UTC, "_id"=>{"$gt"=>BSON::ObjectId('55b0088c5361725886000001')}}]}
  options:  {:sort=>{"a_time"=>1}, :limit=>2}
  class:    Feed::Item
  embedded: false>
```

This is logically incorrect, as the `cursor.criteria` condition is not consistently evaluated when all of the conditions are `$or`ed together.

Note that this appears to be an underlying mongoid issue: https://github.com/mongoid/origin/issues/57

I have written a spec that breaks with the original implementation, and a fix to account for the underlying issue with chaining `$or` conditions within a `.where` query.

Nice work. Please update CHANGELOG via --amend.

Merged via https://github.com/dblock/mongoid-scroll/commit/bdca40018db5395e7da8428835a9c7f00b50008c. Thank you.

I am going to hold cutting a release, not too excited about shipping something compatible only with a beta of Mongoid 5, point to github: 'dblock/mongoid-scroll' for now.

Thanks @dblock!


chore: bump ethers to 5.6.8

Description
Bump ethers package to latest 5.6.8 which fixes a serious BN.js bug incorrectly computing hexstring, see https://github.com/ethers-io/ethers.js/issues/3017
Additionally, package updates adds optimism to networks and provider; plus ENS avatar issues are being fixed resulting in ENS names with avatars now showing up correctly on the leaderboard.
Related issue

Motivation and Context
Upgrade ethers package to latest version
How Has This Been Tested?
Locally build with latest ethers package, checked all pages working correctly, closed an position successfully on OP L2 Mainnet.
Screenshots (if appropriate):

Noticing some weird behaviour on the leaderboard, though. We might need to verify if everything works accordingly.

#891 should be fixed first to avoid the ENS issue, this one is on hold until then.


edit validation.md: use dotnet ef database update in Visual Studio Code section. Use Update-Database in Visual Studio section

^ When the reader selects the 'Visual Studio Code' toggle, the reader sees dotnet ef database update then a use of Update-Database rather than dotnet ef database update.
This commit puts the use of Update-Database in the Visual Studio tab and a corresponding use of dotnet ef database update in the Visual Studio Code tab.
Separately, this commit changes 'Up methods' to 'Up method` because I see only one method in the code block that follows.
Aside
A previous page  uses dotnet-ef rather than dotnet ef.

Internal previews

📄 File
🔗 Preview link

aspnetcore/tutorials/razor-pages/validation.md
Part 8 of tutorial series on Razor Pages

@zipperer, thanks very much for the correction.  Greatly appreciated.  Looking it over..


2019 flutter graphql

2019 flutter graphql

Codecov Report

Merging #18 into WIP will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              WIP      #18   +/-   ##
=======================================
  Coverage   87.05%   87.05%
=======================================
  Files          28       28
  Lines         139      139
  Branches        7        7
=======================================
  Hits          121      121
  Misses         18       18

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19ad724...d67a11d. Read the comment docs.


Change the repository name from "administration" to "documentation"

The issue
The current link for our documentation is https://ontoportal.github.io/administration/, it was relevant there were only the administration section but now that we planned to have multiple sections (for developers, users, home...) we should rename it to "documentation".
By doing that the link will be in the future https://ontoportal.github.io/documentation
Warning
We can't change it, without the explicit approval of @graybeal (and @ontoportal/bioportal) because it will all their referenced links.
How to

Configuration
After the repository name change, we need then to update this config file variables

https://github.com/ontoportal/administration/blob/master/_config.yml#L18
https://github.com/ontoportal/administration/blob/master/_config.yml#L30

Done with https://github.com/ontoportal/documentation/commit/b43bfa3e9f27e8723d5f971aefbebc0997a39b88
After approval of @graybeal  here https://github.com/ontoportal/documentation/issues/10#issuecomment-1402387723


[Feature Request] Theme generator

Is your feature request related to a problem? Please describe.
I like nextui very much, but I've seen there is no theme builder for it, so I started working on one. I made some progress on it and was wondering if you want to mention it in the documentation of nextui. Let me know your thoughts. Thanks!
You can check it here: https://nextui-themegen.netlify.app
Describe the solution you'd like
https://nextui-themegen.netlify.app/
Describe alternatives you've considered
https://nextui-themegen.netlify.app/
Screenshots or Videos
No response

thanks a lot it's amazing

@xylish7 Can you DM me in discord? We'd like to discuss more with you.

Sure! For whatever reason, the discord link doesn't work. My discord username is filipfrincu.

@xylish7 I just sent you a friend request.


457 generate audit logs plus new helper interfaces

Pull Request Overview

Helpers for creating audit logs in Logger class (made the imports easier)
More helpers added to abstract how it's implemented to the calling code
Much more useage of Audit logging added

**REST API Updates **
The /api/v1/audit_logs GET now takes an optional 'module_name' parameter which means that it returns only the audit log options for that module

/rebase

/rebase


fix: webview crypto polyfill issue

Fixes #119
Changes the polyfill to update when onContentProcessDidTerminate is triggered when iOS kills WebView.
Tested using the following script to kill WebView process in the simulator
WEBKIT_CONTENT_PID=$(pgrep -P $SIMULATOR_PID 'com.apple.WebKit.WebContent')
echo "WebKit WebContent PID(s):"
echo $WEBKIT_CONTENT_PID

kill $WEBKIT_CONTENT_PID
echo "WebContent processes terminated."

@im-adithya can you please do a PR in the upstream repo? and why is it called polyfill/index.js? we have multiple polyfills, this will be unclear to developers what it is for, won't it? at least polyfill-crypto? and can you add a TODO at the top explaining this so we don't forget?


Remove the need for a custom property when resolving static web assets from referenced projects

Removes the need for a custom property when resolving referenced static web assets.

This fixes the issue with reevaluating project references. Note this used to take 11s and now its down to 3s for ComponentsApp.Server

@nguerrera @dsplaisted

Can you share the results of msbuild /clp:Summary before and after (without binlog which adds significant overhead)?

@nguerrera sure can do. I collected the binlog from VS. I imagine you are ok if I do the same from command-line? Can this be done with dotnet build or dotnet msbuild?
If not, could you guide me on how to collect the info you want?

Yes command line. Either msbuild or dotnet build as long as it's the same for both before and after.

It looks loke there's also some double evaluation caused by ResolveCustomReferences, which, if I've searche correctly is just a custom task in this repo's infrastructure and not part of the product?

It looks loke there's also some double evaluation caused by ResolveCustomReferences, which, if I've searche correctly is just a custom task in this repo's infrastructure and not part of the product?

I believe so. I'm not ultra-familiar with our build, but we reference things internally through <Reference Include="Microsoft.AspNetCore.Dependency" /> and our build system does the rest to figure out the actual reference.

I'm not totally sure but I believe it'll hit the disk again.

Not sure about what this means. How would you recommend avoiding it?

Yeah, it doesn't respect solution configurations or metadata on the project references.

Could you clarify what do we need to do to support what you describe?

Definitely feels like there's a better way to do this, but I'm not totally clear on what's necessary. Can you describe what needs to be collected?

This feature allows people to define content in the wwwroot folder of their razor class libraries and expose that content to the main application for consumption.
At development time we create a manifest with all the paths from referenced projects and nuget packages that the main web app consumes at runtime, we do this so that you can change the files in your project and see the update on the page without rebuilding the app.
At publish time we simply copy those files to the final place where we expect them to be on disk.
The way this works is:

We resolve the static web assets for the current project, referenced projects and packages.
We produce a manifest as part of the build output that contains all the content roots and the paths they are mapped to at runtime.
We copy the assets to their final destination during publish.

We query the referenced projects for what their assets are, these assets are not static, as they might be the result of running webpack or a typescript compilation, so we query the referenced projects to do whatever work is necessary there and produce the appropriate output.
For the most common case, these assets are whatever is on the wwwroot folder.
Does this help?

:up: :date:
There could have been something wrong with the first time I checked (our build infrastructure customizes A LOT of stuff)
but I've checked again after adding the fixes I think were suggested here and I got it to

Which clearly shows this got fixed.

I had one more comment, which @rainersigwald should check, and I would prefer to let @rainersigwald sign off here.


Warning C4551: function call missing argument list

Using Cython version 3.0a4 with python3.7 on windows7 x64, I received
plugin.cpp(329104): warning C4551: function call missing argument list
#if PY_VERSION_HEX < 0x030600A4 && CYTHON_PEP487_INIT_SUBCLASS
    if (likely(result) && likely(PyType_Check(result))) {
        if (unlikely(__Pyx_SetNamesPEP487(result) < 0)) {
            Py_CLEAR(result);
        } else {
            result = __Pyx_InitSubclassPEP487(result, mkw);
        }
    }
#else
    (void) __Pyx_GetBuiltinName;   <-- line 329104
#endif

Cython version 3.0a1 did not show this warning.
Feel free to close if this isn't relevant.

Oh, great. That line is literally only there to avoid a warning, about the function being unused, specifically. Does it help to take the address of the function instead, i.e. (void) &__Pyx_GetBuiltinName ?

@scoder, sorry for the naive question but should I do the test to see if the change does what you want it to be done? I certainly open for doing it.


Consider making 0ºC / 32ºF y-axis tick label bold on temp/precip charts

The values chosen for the y-axis tick intervals are generated dynamically based on the data being shown, so this may not make sense unless we manually choose static set of y-axis tick labels for all charts.

https://plotly.com/javascript/text-and-annotations/#simple-annotation
This can work by sticking an annotation somewhere at the relevant y coordinate maybe!


fix: Fixes TypeError when password is unset

This is a hotfix for a TypeError which happens in case the password is not set or the user does not exist.
[2023-05-15 17:22:22,914] /.../viur/core/request.py:299 [ERROR] ViUR has caught an unhandled exception!
[2023-05-15 17:22:22,914] /.../viur/core/request.py:300 [ERROR] a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/project-viur3/lib/python3.10/site-packages/viur/core/request.py", line 275, in processRequest
    self.findAndCall(path)
  File "/home/user/.local/share/virtualenvs/project-viur3/lib/python3.10/site-packages/viur/core/request.py", line 623, in findAndCall
    res = caller(*newArgs, **newKwargs)
  File "/home/user/.local/share/virtualenvs/project-viur3/lib/python3.10/site-packages/viur/core/modules/user.py", line 247, in login
    is_okay &= secrets.compare_digest(stored_password_hash, passwd)
TypeError: a bytes-like object is required, not 'str'

I've started a refactor of the login function regarding the res and status variables.


[Feat:] Add Navbar in below tablet view section

Is your feature request related to a problem? Please describe.
The navbar is not good in below tablet view section
Describe the solution you'd like
Add Navbar in below tablet view section
Additional context
Add any other context or screenshots about the feature request here.
Add this feature in below tablet view section

I am GSSOC24 Contributor
please assign me, I would like to work on this issue

@SrinivasDevolper You have been assigned this issue, please start working on it.

Thanks for assing


TC.1.2 The new cards should be displayed among their hand cards

637f09692773a0d4433d5b637a75dbb3ba2e40e4
This was worked on in a coprogramming/codeWithMe setting


2.1 docs sidenav issues

This issue is for tracking feedback about the new 2.1 docs sidenav.

Nothing reported, so closing this.


Crash on startup "could not find libcore in sysroot path"

When I open a .rs file the language server tries to start and fails:
rust-analyzer failed to load workspace: could not find libcore in sysroot path `/home/vscode/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library`

UPDATE: I think I fixed it by running rustup component add rust-src. Maybe the extension could do that automatically?

Rust-analyzer should do that automatically already.

If we got to the "could not find libcore in sysroot path" error message, discover_sysroot_src_dir ran successfully, so /home/vscode/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library existed, but the core directory inside did not. I'm not sure how that would happen though :confused:

Maybe a race condition - it could take a while to download?

I had this issue when vscode installed rust-analyzer when I opened a Rust project. Adding the rust-src like mentioned worked for me.


Scaling / high res support

There are some issues on high resolution displays like below. Is there any way to enable/fix scaling? I found these issues in Ryven 2.4.2 on a 15" 4K screen (X1 Extreme).

Ah yes, I've had these issues too. I am using a few pixel based sizes internally, which messes up the layouts on higher res screens, which I can probably convert to absolute/DPI dependent scale.
Some parts might get difficult, for example the node title is already point sized, but the spacing in the layout doesn't seem correct (and these QGraphicsLayouts are a pure nightmare, I hate them so much, I wouldn't be surprised if this gets really ugly). But I'd like to provide support for any resolutions in general. I only have a full HD screen to test right now, and there's too much stuff in the pipeline currently, but I will try some fixing during the next weeks.

This PR https://github.com/leon-thomm/ryvencore-qt/pull/21 should solve it.

Indeed, thank you!


Added UWP project

It's an awesome sample!. I added the UWP project and the necessary changes in the views to adapt everything to each platform.

Thanks a lot, had this on my todo list! :)


Blank rectangle on Android SDK 19 or lower

Flutter 1.12.13+hotfix.7
Dart 2.7.0
Hi! It's said on plugin description that it works on android SDK 17 or higher, however it just shows a blank rectangle when run over SDK 17, 18 and 19. I've tested on emulator and on physical device.
flutter_inappwebview version is 2.1.0+1
No error is shown on logcat. Indeed, progressbar seems to be working.

Same error here. Blank screen and no error message :-/

Same

Same

I have a very similar error #418 not sure if it is the exact same one.

In general, this plugin requires Android 17+.
For InAppWebView widget is supported only on Android 20+.
Unfortunately, this is a restriction of Flutter API: https://api.flutter.dev/flutter/widgets/AndroidView-class.html
and I can't do anything about that.
Here is an open issue on the official Flutter repo about that: https://github.com/flutter/flutter/issues/23728
I have also updated the README.md about that.


Shell Sort in Python

PROGRAM
(A clear and concise description of what the feature is.)
Have you read the Contributing Guidelines on Pull Requests?
YES
Motivation
Shell sort in Python
Pitch
This program in python is implemented to facilitate coders to code shell sort in Python

can I work on this issue

/assign


ci(build-and-test): fix to work only on the default branch

https://github.com/autowarefoundation/autoware_common/pull/23 don't work. This PR fix it.

But let me test this before merging.
Also, could you fix your commit message, please? See here for the reason.
https://github.com/autowarefoundation/autoware_common/pull/22#issuecomment-1031095967

@h-ohta Since this is a PR from your fork, could you run the workflow here?
https://github.com/h-ohta/autoware_common/actions/workflows/build-and-test.yaml

Worked!
https://github.com/autowarefoundation/autoware_common/actions/runs/1805116363


re-order index

Codecov Report

Merging #5655 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #5655      +/-   ##
============================================
- Coverage     82.84%   82.82%   -0.03%
+ Complexity    11798    11795       -3
============================================
  Files          1392     1392
  Lines         45361    45361
  Branches       5629     5629
============================================
- Hits          37579    37568      -11
- Misses         5160     5172      +12
+ Partials       2622     2621       -1

Impacted Files
Coverage Δ
Complexity Δ

...n/java/io/crate/exceptions/JobKilledException.java
0% <0%> (-66.67%)
0% <0%> (-1%)

...exceptions/GroupByOnArrayUnsupportedException.java
66.66% <0%> (-33.34%)
1% <0%> (-1%)

...crate/executor/transport/TransportShardAction.java
72.97% <0%> (-24.33%)
8% <0%> (-2%)

...o/crate/analyze/SafeExpressionToStringVisitor.java
83.33% <0%> (-16.67%)
7% <0%> (-1%)

...a/io/crate/operation/join/NestedLoopOperation.java
82.85% <0%> (-11.43%)
14% <0%> (-1%)

...rc/main/java/io/crate/analyze/UnboundAnalyzer.java
75% <0%> (-10%)
2% <0%> (ø)

...src/main/java/io/crate/jobs/NestedLoopContext.java
66.66% <0%> (-5.56%)
7% <0%> (-1%)

.../java/io/crate/blob/TransportDeleteBlobAction.java
75% <0%> (-5%)
4% <0%> (-1%)

...rc/main/java/io/crate/blob/BlobTransferTarget.java
81.09% <0%> (-3.05%)
25% <0%> (-1%)

...c/main/java/io/crate/exceptions/SQLExceptions.java
68.62% <0%> (-2.95%)
34% <0%> (-2%)

... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c903fec...7af416b. Read the comment docs.


Add Python 3 support via six

The test suite passes on both Python 2 and Python 3. I have not tested the library yet in production.

Nice. Thank you. I will merge this directly as is.
I don't really see a strict need for the absolute_import's but i does not really hurt either.


Supporting ACME for Cert management

Automatic Certificate Management Environment (ACME) is now an IETF RFC and it is supported by lets-encrypt. It would give the ability to manage certificates without maintenance overhead.

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Spring: Api Objects in methods annotated with @GetMapping @PostMapping

I observed that ApiObjects are only detected in controllers whichs methods are annotated with @RequestMapping.
They are not detected in controllers which are using @GetMapping, PostMapping ... instead.

Hello, I also meet this problem. After debug I found that this is caused by:
AbstractSpringJSONDocScanner class's jsondocObjects method just get methods with RequestMapping annotation.
Hope someone can fix this problem! Thanks.
Below is the AbstractSpringJSONDocScanner class's jsondocObjects method code:
@Override
	public Set<Class<?>> jsondocObjects(List<String> packages) {
		Set<Method> methodsAnnotatedWith = reflections.getMethodsAnnotatedWith(RequestMapping.class);
		Set<Class<?>> candidates = Sets.newHashSet();
		Set<Class<?>> subCandidates = Sets.newHashSet();
		Set<Class<?>> elected = Sets.newHashSet();

		for (Method method : methodsAnnotatedWith) {
			buildJSONDocObjectsCandidates(candidates, method.getReturnType(), method.getGenericReturnType(), reflections);
			Integer requestBodyParameterIndex = JSONDocUtils.getIndexOfParameterWithAnnotation(method, RequestBody.class);
			if(requestBodyParameterIndex != -1) {
				candidates.addAll(buildJSONDocObjectsCandidates(candidates, method.getParameterTypes()[requestBodyParameterIndex], method.getGenericParameterTypes()[requestBodyParameterIndex], reflections));
			}
		}

		// This is to get objects' fields that are not returned nor part of the body request of a method, but that are a field
		// of an object returned or a body  of a request of a method
		for (Class<?> clazz : candidates) {
			appendSubCandidates(clazz, subCandidates, reflections);
		}

		candidates.addAll(subCandidates);

		for (Class<?> clazz : candidates) {
			if(clazz.getPackage() != null) {
				for (String pkg : packages) {
					if(clazz.getPackage().getName().contains(pkg)) {
						elected.add(clazz);
					}
				}
			}
		}

		return elected;
	}

True, meta annotations are not resolved by the getMethodsAnnotatedWith() call, so GetMapping, etc are not found.
Opened an upstream bug ronmamo/reflections#269


Graceful Shutdown

There should be a way to shutdown a queue worker so it finishes its current task before destroying itself

Added in #14


Add plot recipe for solution struct

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/32110
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.2.0 -m "<description of version>" 9bf6b33c537cbf0d802196507bf7cbe88589d8ca
git push origin v0.2.0


Delete Movie/show from kodi menu

Not sure if possible but be handy to have a delete button when selecting a movie or TV show from kodi menu. As in the same as pressing context button, then manage, then remove from library direct from kodi.

I will not add this.  This would have added if a restriced user is added to htpc-manager

I don't think that's possible as long as the user has "allow file naming and deletion" deselected in Kodi? Those without criters have this enabled...

I was 3 crazy critters and I don't think the api supports it. You can use the keyboard to delete the show, press c on your keyboard while the kodi module is open will bring up the context menu


Unable to verify contract on Etherscan (Rinkeby)

I was able to deploy the contract (on Rinkeby) per your latest video tutorial with no issues. When I look at the contract on Etherscan it tells me I need to verify the contract which I attempt to do using the settings on the attached screenshot.

I use the JSON file that Remix generates and paste the constructor arguments in ABI encoded format. To get the ABI encoded constructor arguments I copy the ABI out of Remix and into https://abi.hashex.org then fill out my parameter values to generate the ABI-encoded data. I then paste that into the Constructor Arguments and click Verify and Publish.
I then get the following error (Unknown key "abi"). I have attached a screenshot. Any help appreciated.

I was able to bypass this issue using the Etherscan Contract Verification plugin for Remix. I just needed an API key from Etherscan...there was also about a 10 minute lag between my API being created and it being usable.


Deletion semantics for higher level resources

Federation v2 follows a layered approach such that the higher level resources depend on the lower level federated types for propogation and rconciliation of k8s objects into federated clusters.
To create, for example a automated distribution of deployment replicas, an user can only create a FederatedDeployment and ReplicaSchedulingPreference (as defined in #46). The higher level RSP resource controller will observer this and create an override (FederatedDeploymentOverride) and a placement (FederatedDeploymentPlacement) to reconcile this user intent into clusters. The RSP controller can further update the corresponding override and template to reconcile cluster capacity and/or clusters leaving or joining this federation.
A peculiar case is when the higher level resource is deleted; should the controller also delete the corresponding override and placement resources or leave them behind? This becomes peculiar, because the lower level resources (override and placement) might have existed prior to creation of the higher level resource, in which case the higher level resource, adopted and updated them, if needed.
The answer probably is to use a cascading deleting mechanism and/or a parentRef (ObjectMeta.OwnerReferences) in such a scenario.
The user can specify --cascade=true/false (with some default) to control this behaviour.

When would it ever be desirable for higher-level resource deletion to delete lower-level resources by default?  The dependency graph is arguably directional (from high-level to low-level).  It could be possible for a UI to support manual deletion of related resources, but I think it would be surprising for that to happen automatically.
I do think there is value in the controller for a higher-level type to add finalizers to lower-level types (e.g. scheduling controller add finalizers to template) to ensure cleanup if the lower level resource would be removed.  Such a scheme would be consistent with the directional relationship of the resources.

When would it ever be desirable for higher-level resource deletion to delete lower-level resources by default?

I did not specify a necessary default behaviour :). IMHO, either (cascade or no cascade) could be default.
The use case that we have in mind is:
User creates a FederatedDeploymenttemplate and provides preferences using ReplicaSchedulingPreferences and thats all. In this case the override and placement actually are created by the system. When user doesn't want these anymore, he deletes FederatedDeploymenttemplate ReplicaSchedulingPreferences. Without an option to delete the lower level resources in the dependency graph, the override and placement will be left behind. Its arguable that without a template, override and/or placement both wont be of any use. To me it seems valuable to leave a switch with the user to have control on such behaviour.
Also one view of the layered system would be to visualise the higher level resources to be users of lower level resources, where its up to the higher level resource controller to define its usage behaviour, which includes crud of its target resource in a layered fashion (or the dependency tree, as you call it).
Perhaps this discussion can be widened to considering the template as the primary resource among template, override and placement. Does it make sense?

I think the template is already on its way to being the primary resource.  At present explicit placement is required, but #75 is likely to make that optional for all namespaced resources.  The catch is that an override resource needs to exist before its associated template if member clusters are to receive those overrides on creation of target resources.  This is critical for Jobs, whose Completions field is immutable (#44).
Its not clear to me what deletion semantics should be given the dependencies between resources.  Near-term (certainly pre-alpha), I would not expect automated deletion to be a priority.  Hopefully users can participate in improving the UX once they have an alpha release to work with.

@irfanurrehman Would it make sense for the RSP controller to add ownership references to rsp resources such that deletion of a template would ensure deletion an owned rsp resource?

@irfanurrehman Would it make sense for the RSP controller to add ownership references to rsp resources such that deletion of a template would ensure deletion an owned rsp resource?

Yes it seems logical to me, because creation of the template would be what triggers the creation of placement and override in the first place.


Sidre: Write example and documentation on restarting

Went to show a team how to do Sidre restarts and realized we didn't have any documentation on it or an example.

Some of it is covered in sidre/docs/sphinx/parallel_io_concepts.rst but there's nothing pointing people there saying "This is for restart."  Also it doesn't cover the more complicated usage patterns such as the multi-step process for external data.  An example that really looks like a restart in the middle of a calculation and not just a static save/load equivalence would be good as well.


[vm/ffi] API: Pointer load and store as extension methods

These methods would be better exposed as extension methods:

Pointer.load (this issue)
Pointer.store (this issue)
Pointer.asExternalTypedData (but keep original) https://github.com/dart-lang/sdk/issues/38610
Struct.addressOf https://github.com/dart-lang/sdk/issues/38611

update from @dcharkes: I split this issue into three to keep the CL sizes manageable.

/cc @johnniwinther

Putting this in Q3 stable since extension methods should be usable by then.

Pointer.asExternalTypedData (but keep original)

@sjindel-google, how did you envision this? Instance methods shadow extension methods.

I've made CLs for the API change and updating the tests, samples, and benchmarks. Feel free to leave comments on those.
Note, we cannot land these CLs as long as the extension methods are behind a flag as experiment.

Do we want to keep load and store as @deprecated for a while? Or would we like to remove them right away? If we deprecate them, how long should we keep them? Till Dart 2.7?
Pro for deleting them: We do not have deprecated stuff in the ffi 1.0 release.
Pro for making them deprecated for a while: We don't break existing customers right away.
Maybe we should delete them right away. We have been upfront about API changes, and it will clean up the API and our internal implementation.
What is your opinion @sjindel-google @mkustermann?

I vote for breaking them, but we should land all our breaking changes for 1.0 at once.

we should land all our breaking changes for 1.0 at once.

:+1: Yeah, or we would break people on master and g3 multiple times.

Note, we cannot land these CLs as long as the extension methods are behind a flag as experiment.

@johnniwinther has just turned on the flag by default. We are unblocked. :+1:

/cc @lrhn for his view on .val and .ref.


Support using create menu with initially selected roadpoint

Right now, you can only use the "Create" menu to place in an initial 2x2 road if nothing else is selected, and it always places it at the origin.
The proposed change is to support adding a 2x2 (or whatever is selected) as the "next" road placement if an initial roadpoint is selected (nothing should be possible if more than one are selected). Under the hood, it should treat the 2x2 road to be added as though it's like a scene or prefab to be placed as a whole entity, and all the create menu is doing is spawning in those road points, and connecting the two together.
Ideally the new roadpoint template is always oriented based on the orientation of the currently selected node.

Before starting on this task, we likely need to align first on the general way to approach this and how to refactor current code to accomodate.


Using pretrained models

The paper mentions that you performed end-to-end validation of AlpacaFarm. Do you have the code up on Github for that? I want to use the LLM pre-trained on human preferences to generate some more preferences.

For validation you can check: https://github.com/tatsu-lab/alpaca_farm?tab=readme-ov-file#running-automatic-evaluation or directly use AlpacaEval
This is the reward model from human preferences: https://huggingface.co/tatsu-lab/alpaca-farm-reward-model-human-wdiff
Ans the RLHF model: https://huggingface.co/tatsu-lab/alpaca-farm-ppo-human-wdiff

I've been trying to generate text ising ppo-human but I've just been getting gibberish. It works fine when I use LLama2. Is there an example in AlpacaEval I can refer to?

here are some outputs: https://github.com/tatsu-lab/alpaca_eval/blob/main/results/alpaca-farm-ppo-human/model_outputs.json
decoding configs: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/models_configs/alpaca-farm-ppo-human/configs.yaml


Tool: Transitioning IDIR Usernames from Email-Based to GUID-Based

Keycloak Username Issue with IDIR Users
Problem:
Context: Current IDIR user accounts have usernames that are based on their email addresses. As noted by Marco and confirmed by the IDIM team, email addresses and IDIR IDs associated with users' IDIR accounts are recycled, leading to cases where a different user could have a recycled email or IDIR ID.
Forms/submissions ownership in DGJ for IDIR users is currently based on the email-based username. Therefore, in the event that an email is reused, this causes two issues:

Privacy Issue: Users with a recycled email will have access to all the forms/submissions of the previous IDIR user with that email.
If the previous email holder has an existing IDIR account in DGJ Keycloak, Keycloak will have difficulty separating these two IDIRs with the same email. Keycloak will see both IDIRs as the same and try to merge them, leading to issues. This has caused some login issues for users and would require manual resolution.

Solution:
Context: The SSO team suggests moving to a GUID-based username as the GUID is the only unique identifier.
Potential solutions:

Migrating IDIR users' usernames from email-based to GUID-based.
Extending Keycloak with extra logic to handle the case when two IDIRs with the same email emerge in the system.

Why Was Email-Based Username Chosen and What Was the Reason?
At the time of employee submission of forms that require a manager's review (e.g., Telework and Mat/Pat), the only identifier for the manager is the email that the employee adds or that is auto-populated at the time of form submission. We need to add a manager role to that manager. If the manager has already logged into DGJ, we look for their user account and add the manager role. If the manager has never logged into DGJ, we need to create a temporary Keycloak user account and assign the manager role to it. Having the email as the only identifier, we decided to architect the IDIR users to have their email as their username so that we can create a temporary user account for the manager, constructing a username based on their email and adding the necessary roles to it. Later, when that manager logs in to DGJ for the first time, their actual information will be merged with the temporary account.
To switch to GUID-based usernames for IDIR users, we need to find a way to add the manager role to managers who are added to forms but haven't logged into the DGJ platform at that time. One possible solution is to obtain the manager's GUID based on their email from PeopleSoft at the time of employee submission. This needs to be confirmed by the ODS team for feasibility. Another option, as suggested by Marco, is to trigger the logic for adding the manager role when the manager clicks on the link sent to them in the email.
Other System-Wide Implications if We Decide to Switch to GUID-Based Username for IDIR Users:

Keycloak:

Changing all the current IDIR users' usernames to GUID-based usernames.
Changing all the temporary IDIR user accounts for managers who haven't accessed the system.

Formio:

Changing the ownership identifier for submissions and forms from email-based to GUID-based for all IDIR users.

Camunda:

Changing all task assignments for IDIR users from email-based to GUID-based. Several tables would require this change.

API:

Changing the ownership of drafts/applications/new-version-feature-notification/form-process-mappers from email-based to GUID-based for all IDIR users.
Changing the related logic that allows/disallows IDIR users to access forms.

Website:

Changing the related logic that allows/disallows IDIR users to access forms.

Please note that the above migration needs to be done for each environment. This means there will be a temporary outage for a few hours in the PROD environment to carry out the migration.
Status of Other IDPs (BCeID, BCSC, and LDB):
We have configured other IDPs to rely on the most unique identifier as the username, and they won't have the issue of email-based usernames as IDIR users:

BCeID is configured to rely on BCeID username, which is a unique ID as the username ${CLAIM.bceid_username}_bceid.
BCSC is configured to rely on BCSC GUID ID as the username. Technically, it is recognized as the token sub-ID ${CLAIM.sub}_bcsc.
LDB is configured to rely on LDB UID as the username. It is set internally at the LDB level and passed under ldbuid to DGJ Keycloak.

@T854236
Context: The goal is when an employee fills a form, getting their manager's GUID. The employee might put any employee's email in the manager's field (not necessarily the one prepopulated from ODS)
Question: Is it possible to get the manager's GUID of an employee? Other words, can we have an endpoint to search for an employee by email and get their GUID?

Hi @iman-jamali-fw,  the employee_demo endpoint does contain email and will allow you to filter by email to get the GUID.  I believe this dataset is now part of your caching and might be easy for you to search locally vs. hitting the endpoint.

@simensma-fresh  Please feel free to edit/correct the details in the findings above.

I have reached out to Dennis (TES dev on the PDP team) to inquire what solution was put in place for the PDP application. Hoping we can connect him with our dev team soon.

<EMAIL_ADDRESS>this is the ticket in question for the DJ team

@MeghanStothers @Abuchana @Stella-Archer cc @T854236
Seems like EMPID (employeeId) is the most unique identifier and won't be recycled. However, not all employees have EMPID. Are there cases in which DGJ gives service to an IDIR user that doesn't have EMPID?

Potential issues that have been learned so far for IDIR user's identifiers.

email:

Can be recycled. Can cause privacy issues. A Different user with a recycled email can access the previous user's data in DGJ.
User can request to change it. Same IDIR when email is changed, no longer see their data in DGJ.

GUID:

Can change (i.e., when an employee moves to another ministry). Even if email stays the same, that user will lose access to data in DGJ.

EMPID:

Protected data. Should be checked to see if it can be stored in the Keylcoak and the DGJ system.
No all employees have EMPID. Accessibility issue

@iman-jamali-fw and @warrenchristian1telus please confirm following update is accurate and complete, edit & update if necessary.
UPDATE:
Team met with Dennis Cinco from the PDP to see what we could learn from how they are approaching the issue
High level findings:

PDP was designed to use GUID but ran into issues and so now use EMPID to check against GUID
DGJ has a broader audience that does not necessarily have EMPID
PDP has access to EMPID because of the unique need for tightly tied employee/supervisor relationships
DGJ also has many forms needing tight employee/supervisor pairings

Action/Next Steps:

[ ] Warren to investigate getting access for the team to the GUID look up table from Mohan
[ ] Iman to work towards plan to migrate off email authentication
[ ] DGJ Team will use employee GUID and check against EMPID to confirm user
[ ] DGJ Team develop plan for cases when GUID fails and there is no EMPID to check against

Sounds good to me @Stella-Archer .
A couple of updates.
We now have access to the GUID lookup endpoint for dev / test, and prod is scheduled for implementation next week (26th).
@iman-jamali-fw FYI, it’s pretty big. Big enough to possibly crash your browser if you test it that way (80+ MB), and pretty slow. Much more reasonable to just get the relevant fields by appending the select param to reduce some overhead (2.6MB):
?$select=EMPLID,GUID

https://analytics-testapi.psa.gov.bc.ca/apiserver/api.rst#Datamart_Telework_peoplesoft_guid_mapping
https://analytics-testapi.psa.gov.bc.ca/apiserver/api.rsc/Datamart_Telework_peoplesoft_guid_mapping

Due to workflow from PeopleSoft > BI > ODS / CDATA production data updates (such as a new GUID) will take up to 48 hours to propagate to (CDATA).
I’ve raised the issue with the team, and am awaiting feedback / discussions on a more timely solution.

Following discussions with the PDP team and internal technical experts, we've outlined the next steps:
Step 1: Transition from Email-Based to GUID-Based Usernames for IDIR Users
Background: Currently, DGJ uses two types of usernames - Keycloak-level usernames, which are based on IDIR information, and system-level usernames, which grant access to their DGJ contents and services (e.g., formio, API, Camunda). Both of these usernames are currently tied to the user's email address associated with their IDIR.
Why the change? Using email-based usernames can lead to privacy concerns due to email recycling. GUIDs associated with IDIR users have proven to be unique identifiers.
Our solution: We'll migrate system-level usernames to a custom GUID generated by Keycloak, known as the Keycloak user ID. This ensures that even if a user's attributes change in the future, their internal DGJ ID remains unique. For example, if an existing IDIR user's IDIR and GUID change (e.g., due to a ministry change), we can update their Keycloak user account while preserving their unique ID, ensuring continued access to DGJ content.
In summary, we'll migrate IDIR users from email-based usernames to IDIR GUIDs at the Keycloak level, while system-level usernames will transition to Keycloak's user account ID.
The required details for implementation will be added to ticket #1357.
Step 2: Automate Handling Exception Cases When an Existing IDIR User's Information Changes
Improving Step 1:
Step 1 addressed privacy concerns and IDIR changes, but it lacked automation. When an IDIR user's GUID changes, verifying their connection to an existing DGJ account requires manual intervention. Consider this scenario: UserA submits a form in DGJ, then switches ministries, gets a new IDIR, and logs in again, appearing as a new user. To regain access, the user must contact DGJ for verification, often involving cross-checking employee IDs.
Related findings:

Keycloak's default login flow offers limited control. Extending it with custom Java classes provides full control over various login flows, not just for IDIR but also for future use cases, like linking IDIR and BCSC accounts.
EMPID serves as a unique and unchanging identifier, retrievable through PeopleSoft API calls and storable in Keycloak user attributes.
Email addresses associated with IDIR accounts can be recycled, potentially requiring additional information from external teams.

By storing EMPID and utilizing API calls and external data, we can design a custom Keycloak login flow to automate exception handling, eliminating manual steps in Step 1.
To design this flow, we'll identify and propose solutions for possible cases. Here are some examples:

When a new IDIR user with a new GUID logs in:

If they have an 'EMPID,' check for matches with existing IDIR users' EMPID:

If matched, update the existing user's account with new IDIR information.
If not, create a new Keycloak user account.

If a new IDIR user lacks an 'EMPID,' check for matches based on email:

If no match, create a new Keycloak user account.
If a match, further validation may be needed through external team data.

What to do between the time an employee's IDIR information changes (such as GUID) but their information hasn't been updated on PeopleSoft

The complete list along with the required details for implementation will be added to ticket #1358.

@bhumin-fw @simensma-fresh @warrenchristian1telus
This ticket is ready for your peer review. Please let me know if anything else needs to be added to the result of the exploration and next steps. Details for implementation and a complete list of cases can later be added to the implementation tickets (#1357 and #1358).

Thanks @iman-jamali-fw - after an email with Marco, it seems we may be able to get current EMPLID, GUID and email / IDIR from BCeID SOAP service (this would need to be a secondary request upon login, or batch sync job). We're in the process of setting up a meeting to discuss our options.
I also discussed this with Phil, and we may be able to get EMPLID set in AD and limited to unique values to keep things stable. Again, pending further discussions.
If we can get EMPLID into AD / KeyCloak, that should solve all our exception cases at once. If EMPLID needs to be unique in AD, it should eliminate (re)creation of users when they switch ministries, etc.. It would also ensure there is never a 24-48 hour delay for things to [re]sync from PeopleSoft to ODS/CDATA.

Excellent summary and research on this one, everyone! We agreed next step is a conversation with TES and other stakeholders to share and take steps.
@warrenchristian1telus Is your last comment the most recent one? You'd mentioned a a "synchronization" solution and I want to make sure we have all possibilities to share with everyone. If not, pop a few notes down. Thanks

Updated information has come in from OCIO regarding SSO.
"If the attribute exists in IDIR AD then we can pass it as an identifier through SAML from SiteMinder.  Keycloak’s integration with us as a defined set of generic attributes.  Since most applications will not require the EMPLID attribute it is not included in the Keycloak payload from SiteMinder."
This suggests that they can enable EMPLID for us, which should solve most of our immediate authentication issues arising from a GUID change.
Our alternate plan is to implement a pipeline from PeopleSoft to CDATA. We would need to hook into the GUID update process, and have it send any updates to CDATA, where it could then be consumed by each application.
My preference is to go with the first option. As long as it can solve our issues within an acceptable time. This should resolve the issues without as much effort, although it would likely still require us to do some sort of batch job to synchronize existing accounts with new data (GUID to EMPLID).

I reopened this one to keep it on our radars for something we'll need to action after mat/par deployment and (before?) Telework deployment and new application upgrade

The team will discuss this work this morning.

@warrenchristian1telus would this be a good thing to tackle when we are doing the cData migration? I'd like to get some sense of when would be the ideal time to look at this
@MeghanStothers  is this also some thing that needs to be confirmed at other teams  'all products' level

@Stella-Archer I think this will depend a bit on what changes will be required. Will we need to change the structure of data in CDATA, or are all the changes on the application side?
Either way, I doubt it would be a big impact to the CDATA migration. We just need to ensure that any changes are identified and verified in the appropriate environments.


MudList/NestedList does not honor Dense property

Bug type
Component
Component name
MudList
What happened?
When specifying Dense="true" for MudList, the items in the NestedList are not dense.
Expected behavior
NestedList items should be dense.
Reproduction link
https://try.mudblazor.com/snippet/QYwGYVOXUVrVQnlQ
Reproduction steps

Create MudList
Set Dense="true"
Add MudListItem
Add NestedList in the item
Add MudListItem in the NestedList

Relevant log output
No response
Version (bug)
6.0.11
Version (working)
No response
What browsers are you seeing the problem on?
Microsoft Edge
On what operating system are you experiencing the issue?
Windows
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

In the meantime you can put it on the MudListItem, that'll work.
<MudListItem Dense="true" ...


CDN Availability

Added project link to jsDelivr CDN website.  A bot will update within an hour when a new release is tagged.
Not sure if you want all the fancy aliasing & concocting listed.

@tomByrer can you include all variants, including -next and -lite?

I modified update.json https://github.com/jsdelivr/jsdelivr/commit/3704f917f0794e85b125d9cef659ffb5e2eeeb48
All future versions will be added as you requested. Do you want me to manually add these files to the latest as well?

Thanks for your help - we've now featured jsDeliver as part of the Web Animations Resources update. However, we don't want to list a specific CDN in this repository, so I'm closing this PR.


bump dompdf/dompdf version to 0.8

Because of dompdf/dompdf/issues/1272 i was trying to require the latest release of dompdf/dompdf, which is not possible because of "dompdf/dompdf": "^0.7" instead of `"dompdf/dompdf": "~0.7".
So i raised the required version of dompdf here :-)

Coverage remained the same at 79.07% when pulling a0a3c7acc9193ea719005b315a6ef06da495d5c1 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Coverage remained the same at 79.07% when pulling a0a3c7acc9193ea719005b315a6ef06da495d5c1 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Coverage remained the same at 79.07% when pulling a0a3c7acc9193ea719005b315a6ef06da495d5c1 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Coverage remained the same at 79.07% when pulling a0a3c7acc9193ea719005b315a6ef06da495d5c1 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Coverage remained the same at 79.07% when pulling ffdc378c9d6f6cef8b8bc1aceca146770d55ae15 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Coverage remained the same at 79.07% when pulling ffdc378c9d6f6cef8b8bc1aceca146770d55ae15 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Coverage remained the same at 79.07% when pulling ffdc378c9d6f6cef8b8bc1aceca146770d55ae15 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Coverage remained the same at 79.07% when pulling ffdc378c9d6f6cef8b8bc1aceca146770d55ae15 on 84m:patch-1 into 120503b0c889192e15fc3d8f941afb63b862a8fc on core23:master.

Thanks @84m
There are a lot of BC breaks since the last major. I plan to release a new major release by the end of the month.


Embed connection stall reason in "Loading" div element for traceability of Playwright stalls

Might be worth surfacing more clearly to users in the future, but one thing that @iterion raised when I asked about potential engine issues with our CI Playwright tests is that we have no insight to what backend issue is causing each of these "loading stalled" tests, because they appear in our logs as simple errors like this:
locator resolved to visible
…

But if we included a reason in that div as a data-* attribute it would be easier to correlate with what went wrong.

Nice!
specifically what information do we put in here, we could just JSON dump all of the network state data?

Engine returns nice errors like too many connections we should just surface those like to the user as well


Add print css

What is the context of this PR?
Adds print css which styles the page for printing.
How to review

Go into the page template macro and change the media attribute in this line <link rel="stylesheet" media="print" href="{{ assetsURL }}/css/print.css"> to "screen".
Then check all components look as you would expect if you were to print.

 Print style sheets

Quick list of some things I spotted:

Main nav displaying as a list
No url displaying with links
Summaries are a bit squashed
Hub and spoke breaks the print page so will print on page 2
Hero requires additional work

Accordions and collapsible need to be expanded (basically matching the no js styling)

All tables need to fit on the page


[css] refine vendor-specific warnings in vendor-specific rule blocks

VS Code Version: 1.72.2

Please see the picture. VSC will prompt when a CSS property exists only with vendor prefix, which is a nice feature. But if the entire rule block is vendor-specific (the vendor-specific pseudo element), maybe there is no need to prompt.

@aeschli I'll submit a PR soon.

@aeschli Made the PR. I'm not sure this is a good solution though.

@babakks Hi, as far as I know, Firefox supports the -webkit- prefix, so it is possible to use a -webkit- attribute inside a -moz pseudo element (technically), the code below will work.
input[type="range"]::-moz-range-thumb {
    -webkit-mask-image: linear-gradient(to bottom, transparent 50%, black 50%);
}

@LIXiangChen Thanks for reminding this, but I think such things relate to the way CSS data are covered by the extension. I'm not sure about the policy on that side, though.


AssertionError: No. communities (...) differ from no. nodes (...)

I noticed that for a 0-based graph with nodes from 0 to n-1,
the script doesn't work properly and throws the following error:
AssertionError: No. communities (9705) differ from no. nodes (9706)
I used the script with the command:
julia CGE_CLI.jl -g graph.edgelist -e ./graph/node_embeddings/node2vec-d-128_l-80_r-18_k-16_e-1_p-1.0_q-1.0/nodes.emb -v
The verbose output is:
2 columns and 27403 rows in edgelist file.
Graph contains 9706 vertices
Done preparing edgelist and vertices weights
0.976024

I noticed, that the communities file generated with the Louvain algorithm ends like this:
9702 0
9703 1
9704 2
9705 1

So the script mistakenly assumes in the Assertion that not all Nodes are assigned to a community,
while this is actually the case because the nodes start at 0.
For me changing the graph to 1-based solved the problem.
But I guess it would be nice to fix it or at least to mention in the readme that for automatic community generation a 1-based graph is required.

Thank you for submitting the issue. Indeed, the implementation was handling the automated clustering generation only for 1-based vertices. For some reason Louvain binary always generated cluster assignment for node '0', even if there is none in the edgelist. After the fix in v2.0.1, clustering function checks how the labelling is set up and modify the output file accordingly.


k0sctl: 0.12.6 -> 0.13.0

Description of changes
https://github.com/k0sproject/k0sctl/releases/tag/v0.13.0
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

❯ nix-build -A k0sctl
/nix/store/zdcwsik1sgckb9awzqzrq0r50ygh7sdy-k0sctl-0.13.0

❯ /nix/store/zdcwsik1sgckb9awzqzrq0r50ygh7sdy-k0sctl-0.13.0/bin/k0sctl version
version: 0.13.0
commit: HEAD

@ofborg build k0sctl


add develop option to Makefile

add develop option to Makefile
Description
Add a develop option to the Makefile to start the container and attach to stdout.
Related Issue
#9
Motivation and Context
See #9
How Has This Been Tested?
Run make develop
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

Could you please add a Development section in the readme?


🐛 label reference

I think this workflow is not working correctly, it's only fetching the branch so recompiling was is there already..

Ready to 🚢.


Feature request: Support for lateral joins

Is your feature request related to a problem or challenge?
Here is DuckDB's documentation.
The first example given in the above DuckDB documentation is
SELECT *
FROM range(3) t(i), LATERAL (SELECT i + 1) t2(j);

Which gives

i
j

0
1

2
3

1
2

If I try the same in datafusion-cli 37.0:
SELECT * FROM unnest(range(3)) t(i), LATERAL (SELECT i + 1) t2(j);

(Note that I had to wrap range(3) in a unnest, as Datafusion does not yet have a range table function.)
This gives
Schema error: No field named i.
I would have expected that Datafusion would let (SELECT i + 1) t2(j) refer to i from the first table.
Describe the solution you'd like
Support for lateral joins.
DuckDB mentions that the LATERAL keyword is optional.
The first implementation in Datafusion would not need this.
Describe alternatives you've considered
No response
Additional context
No response

Are correlated sub-queries a prerequisite? https://github.com/apache/arrow-datafusion/issues/6140

Hi @aalexandrov  -- no one is actively working on this as far as I ca tell.

IIUC the right way to approach the problem here is to implement better (ideally full) query decorrelation as described in this paper.

I am not familar enough with lateral joins to be sure without some more research
However, to the best of my knowledge current industrial best practice for subquery execution is to decorrelate them to joins (and not actually try and rerun them). Thus if supporting lateral joins in datafusion means supporting more decorrelation rules I think that would be a good design

I am not familar enough with lateral joins to be sure without some more research

The LATERAL join syntax from Postges^1 (which is adopted by DuckDB) effectively is a syntactic counterpart of the dependent join from the Neumann and Kemper paper cited above.

I am not familar enough with lateral joins to be sure without some more research

The LATERAL join syntax from Postges1 (which is adopted by DuckDB) effectively is a syntactic counterpart of the dependent join from the Neumann and Kemper paper cited above. I think the first step is fixing the SqlToRel to make earlier FROM bindings available to the LATERAL input.
Footnotes

https://www.postgresql.org/docs/16/sql-select.html ↩

Thanks -- I have had that paper on my reading list for a while. I think it is time to bump it to the top

I opened a PR (#11456) for the planner extensions required for this.

IIUC the right way to approach the problem here is to implement better (ideally full) query decorrelation as described in this paper.

BTW I finally got around to reading this paper the other week. DataFusion does not implement the fullly general decorrelation alluded to in that paper, but instead implements the "basic TPCH style" decorrelations for most subqueries (aka the basic correlations).
Thus I suspect some of the lateral queries might "just work" once a plan is created for them

There is some more discussion about this here: https://github.com/apache/datafusion/issues/13659#issuecomment-2541722186
Summarizing: Postgres appears to plan this type of query as a NestedLoops join:
postgres=# create table unnest_test (c1 int[]);
CREATE TABLE
postgres=# insert into unnest_test values (ARRAY [1,2,3]);
INSERT 0 1
postgres=# select * from unnest_test u, unnest(u.c1);
   c1    | unnest
---------+--------
 {1,2,3} |      1
 {1,2,3} |      2
 {1,2,3} |      3
(3 rows)

 Nested Loop  (cost=0.00..295.60 rows=13600 width=36)
   Output: u.c1, unnest.unnest
   ->  Seq Scan on public.unnest_test u  (cost=0.00..23.60 rows=1360 width=32)
         Output: u.c1
   ->  Function Scan on pg_catalog.unnest  (cost=0.00..0.10 rows=10 width=4)
         Output: unnest.unnest
         Function Call: unnest(u.c1)

🤔
I think the postgres join operator re-evaluates the right input (pg_catalog.unnest) on each input row to the join which is how the lateral joins can refer to the outer query
The current DataFusion join operators don't run the input over and over again -- they normally run each input and then combine them (they don't restart inputs)
So in a query like
select * from t1, unnest(t1.c1)

I think we would need a (new) type of JoinExec that actually does run the outer query (aka unnest(t1.c1) for each input row of t1) 🤔

Summarizing: Postgres appears to plan this type of query as a NestedLoops join:

PostgreSQL is able to decorrelate some queries, but isn't necessarily the best at it (at least last time i checked some ... few years ago when adding correlated IN support to Trino).

I think we would need a (new) type of JoinExec that actually does run the outer query (aka unnest(t1.c1) for each input row of t1)

Or have Unnest relational operator do this.
This is how Trino does this. Because it doesn't support "join operator re-evaluates the right input", it has to model things differently, which happens to be more performant (as long as you are able to decorrelate).

PostgreSQL is able to decorrelate some queries, but isn't necessarily the best at it (at least last time i checked some ... few years ago when adding correlated IN support to Trino).

Yeah, DataFusion can only run joins that are decorrelated (it has no operator to run the classic subquery where it re-evaluates the subquery on each row)


How can I export table to excel when I use ant-design-pro

The project I am developing is using the antd pro, I need to export the table component to excel .
How could I do this by the use of SheetJS? Could you help me in detail? Thanks.

Assuming you are referring to
https://ant.design/components/table/#header , you can export the generated table or you can export the source data.
To export the generated table, use table_to_sheet on the TABLE DOM element to generate a worksheet.
To export the source data, you can call json_to_sheet on the array of JS objects to generate a worksheet.

Cool! Thank you. So helpful website,but if I only want to use the function of exporting table to excel, need I use the all xlsx or is there a new-hand introduce ? Besides, could you tell me how to get the table element like var elt = document.getElementById('data-table'); in antd?

Yes! What I want to do is the export to xlsx function in the link "http://sheetjs.com/demos/table.

Where could I see the source code ? Please give me a link, thank you

@m26597 that's a question better addressed to the ant developers.  You need to get a reference to the generated TABLE.
P.S.: the source to the page is unminified.  You can view source

I've read the source code through F12, so the last problem is "need I use the all xlsx or is there a new-hand introduce ? Besides, could you tell me how to get the table element like var elt = document.getElementById('data-table'); in antd?"

All right, I would try to realize the function. Anyway, Thank you very much.

I'm sorry, I have another issue, Is the exported excel a similar excel but actually  a text or html just like a excel?

you can try to see
https://github.com/dave-wind/excel