WIP: Add import mode to job reconciler

What type of PR is this?
/kind feature

What this PR does / why we need it:
Allow for existing jobs to be imported into Kueue by disabling suspension/stopping of jobs that do not have admitted Workloads. This allows Kueue to create Workload objects and quota to be reconciled for existing workloads. Intended to facilitate the installation of Kueue into a running cluster.
Example steps for adopting Kueue:

Ensure all new jobs in a cluster contain kueue labels.
Create all ClusterQueues, etc.
Run Kueue in import-mode.
Wait for all non-labeled jobs to finish.
Validate expected Workload admission statuses exist.
Switch Kueue into non-import-mode.

Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?

@mimowo wdyt?

/ok-to-test

@mimowo wdyt?

I like it in principle, and I think this is going to be important. I would like to ask some questions to clarify the process though:

Ensure all new jobs in a cluster contain kueue labels.

which kueue labels concretely, just queue-name or some other too? This entails a separate script I guess which iterates over all jobs and patches them, right?

Create all ClusterQueues, etc.
Run Kueue in import-mode.
Wait for all non-labeled jobs to finish.

How we would have non-labeled jobs since we added the labels in step (1.)

Validate expected Workload admission statuses exist.
Switch Kueue into non-import-mode.

Additional, extra question: do we need to deal / how do we deal with overcommitted resources? Meaning, say someone has running jobs exceeding the quota configured in point (2.).
I see three options:

import them all and temporarily run overcommited,
import them all and preempt some to get under quota
import gradually, never exceeding the quota

My preference would be for (1.) as this is a special mode anyway, but would like to clarify. This may actually require a small KEP to discuss pros / cons. But overall, +1 for the feature.

which kueue labels concretely, just queue-name or some other too? This entails a separate script I guess which iterates over all jobs and patches them, right?

How we would have non-labeled jobs since we added the labels in step (1.)

I guess there are multiple paths that could be taken, I was describing one where there is a system that is responsible for submitting new jobs, and that system is first updated to start submitting jobs as if Kueue were in place. All labels for identifying queues, pod group sizes, etc. would be included even though Kueue is not yet in place. You could also run a script to label running workloads, but my description was assuming that you would just allow those workloads to age out before running Kueue. A user could take either or both approaches (age-out / script-label).

Additional, extra question: do we need to deal / how do we deal with overcommitted resources? Meaning, say someone has running jobs exceeding the quota configured in point (2.).

My original thought was the following which sounds like your (1.):
Allow Kueue to create Workloads for all running jobs during import mode. While suspension/stopping is disabled, some of these Workloads would have an admitted condition and some would not, but all would continue to run. The administrator could then inspect which of the running Workloads are considered admissible under the new Kueue system they are introducing. They could then turn off import-mode and allow the Workloads in violation to be evicted as per the quotas they have defined in the queues - or they could change their queues until all workloads fit in their queues.

I guess Kueue would still mark a Workload with Eviction=true even if it doesn't actually stop the jobs/pods. So an administrator could inspect Workload objects and make sure no critical jobs are marked for eviction before disabling import mode.
Could you confirm this?

Could you confirm this?

Yes, I will test this now.

I was describing one where there is a system that is responsible for submitting new jobs, and that system is first updated to start submitting jobs as if Kueue were in place. All labels for identifying queues, pod group sizes, etc. would be included even though Kueue is not yet in place.

got it, thanks for explaining.

My original thought was the following which sounds like your (1.):
Allow Kueue to create Workloads for all running jobs during import mode. While suspension/stopping is disabled, some of these Workloads would have an admitted condition and some would not, but all would continue to run.

sgtm, thanks for explaining, this is what I would expect, but we should test this manually, and some integration tests would be nice as part of the PR.

@alculquicondor A quick manual test appears to confirm the behavior of the Eviction condition being added but not acted upon (and resources not freed up for the preemptor job to use - while under importMode: true).
I can start to add some automated tests.
➜  jobs git:(import-mode) ✗ k get workloads
NAME                     QUEUE        ADMITTED BY     AGE
job-p1-job-lvhfn-a62d1   user-queue                   5m40s
job-p2-job-sr7rp-badd1   user-queue   cluster-queue   10m
➜  jobs git:(import-mode) ✗ k get workloads job-p2-job-sr7rp-badd1 -oyaml | grep -B 3 Evict
    message: Preempted to accommodate a higher priority Workload
    reason: Preempted
    status: "True"
    type: Evicted
➜  jobs git:(import-mode) ✗ k get pods
NAME                 READY   STATUS    RESTARTS   AGE
p2-job-sr7rp-gbwlt   1/1     Running   0          11m
p2-job-sr7rp-kh9xv   1/1     Running   0          11m
p2-job-sr7rp-lw5rw   1/1     Running   0          11m
➜  jobs git:(import-mode) ✗ # Set importMode: false ...
➜  jobs git:(import-mode) ✗ k edit cm -n kueue-system kueue-manager-config
configmap/kueue-manager-config edited
➜  jobs git:(import-mode) ✗ k delete pods -n kueue-system --all
pod "kueue-controller-manager-574bffb9cf-xcjgv" deleted
➜  jobs git:(import-mode) ✗ k get pods -n kueue-system -w
NAME                                        READY   STATUS    RESTARTS   AGE
kueue-controller-manager-574bffb9cf-rbtdn   1/2     Running   0          10s
kueue-controller-manager-574bffb9cf-rbtdn   2/2     Running   0          13s
^C%
➜  jobs git:(import-mode) ✗ k get workloads
NAME                     QUEUE        ADMITTED BY     AGE
job-p1-job-lvhfn-a62d1   user-queue   cluster-queue   7m20s
job-p2-job-sr7rp-badd1   user-queue                   12m
➜  jobs git:(import-mode) ✗ k get pods
NAME                 READY   STATUS        RESTARTS   AGE
p1-job-lvhfn-c4zzg   1/1     Running       0          12s
p1-job-lvhfn-ll89q   1/1     Running       0          12s
p1-job-lvhfn-xmprj   1/1     Running       0          12s
p2-job-sr7rp-gbwlt   1/1     Terminating   0          12m
p2-job-sr7rp-kh9xv   1/1     Terminating   0          12m
p2-job-sr7rp-lw5rw   1/1     Terminating   0          12m

@alculquicondor A quick manual test appears to confirm the behavior of the Eviction condition being added but not acted upon (and resources not freed up for the preemptor job to use - while under importMode: true).
I can start to add some automated tests.
➜  jobs git:(import-mode) ✗ k get workloads
NAME                     QUEUE        ADMITTED BY     AGE
job-p1-job-lvhfn-a62d1   user-queue                   5m40s
job-p2-job-sr7rp-badd1   user-queue   cluster-queue   10m
➜  jobs git:(import-mode) ✗ k get workloads job-p2-job-sr7rp-badd1 -oyaml | grep -B 3 Evict
    message: Preempted to accommodate a higher priority Workload
    reason: Preempted
    status: "True"
    type: Evicted
➜  jobs git:(import-mode) ✗ k get pods
NAME                 READY   STATUS    RESTARTS   AGE
p2-job-sr7rp-gbwlt   1/1     Running   0          11m
p2-job-sr7rp-kh9xv   1/1     Running   0          11m
p2-job-sr7rp-lw5rw   1/1     Running   0          11m
➜  jobs git:(import-mode) ✗ # Set importMode: false ...
➜  jobs git:(import-mode) ✗ k edit cm -n kueue-system kueue-manager-config
configmap/kueue-manager-config edited
➜  jobs git:(import-mode) ✗ k delete pods -n kueue-system --all
pod "kueue-controller-manager-574bffb9cf-xcjgv" deleted
➜  jobs git:(import-mode) ✗ k get pods -n kueue-system -w
NAME                                        READY   STATUS    RESTARTS   AGE
kueue-controller-manager-574bffb9cf-rbtdn   1/2     Running   0          10s
kueue-controller-manager-574bffb9cf-rbtdn   2/2     Running   0          13s
^C%
➜  jobs git:(import-mode) ✗ k get workloads
NAME                     QUEUE        ADMITTED BY     AGE
job-p1-job-lvhfn-a62d1   user-queue   cluster-queue   7m20s
job-p2-job-sr7rp-badd1   user-queue                   12m
➜  jobs git:(import-mode) ✗ k get pods
NAME                 READY   STATUS        RESTARTS   AGE
p1-job-lvhfn-c4zzg   1/1     Running       0          12s
p1-job-lvhfn-ll89q   1/1     Running       0          12s
p1-job-lvhfn-xmprj   1/1     Running       0          12s
p2-job-sr7rp-gbwlt   1/1     Terminating   0          12m
p2-job-sr7rp-kh9xv   1/1     Terminating   0          12m
p2-job-sr7rp-lw5rw   1/1     Terminating   0          12m

It's a curious behavior. If the preemptor job has higher priority than the evited job, what happens?

One more conceptual question:
Context: IIUC during the import, we create workloads, along with their admission to resource flavors, while the jobs are running.
Is it actually always possible to reconstruct the workload's PodSet admission based on the running pods? Mappoing PodSetAssignment resourceName (say CPU) to resource flavor (given the already running pods), see here: https://github.com/kubernetes-sigs/kueue/blob/bd2d09c287fe6ba6efd2991de741209972f8d9d4/apis/kueue/v1beta1/workload_types.go#L91C2-L91C84.
Say, a job might be running pods scattered across nodes corresponding to different flavors (because there was no specific nodeSelector), so there might not always be a reliable way to create the PodSetAssignments (IIUC). If my understanding is correct, could this be an issue, or we don't rely on that?

/close
in favor of https://github.com/kubernetes-sigs/kueue/tree/main/cmd/importer


Unable to find a working version of libraries. Sharp and aws-cdk does not support Node-16, Node-18 error out with aws-sdk not found.

2024-01-15T09:15:25.547Z	undefined	ERROR	Uncaught Exception 	{"errorType":"Error","errorMessage":"Could not load the \"sharp\" module using the linux-x64 runtime
Possible solutions:
- Please upgrade Node.js:
    Found 16.20.2
    Requires ^18.17.0 || ^20.3.0 || >=21.0.0
- Consult the installation documentation:
    See https://sharp.pixelplumbing.com/install","stack":["Error: Could not load the \"sharp\" module using the linux-x64 runtime","Possible solutions:","- Please upgrade Node.js:","    Found 16.20.2","    Requires ^18.17.0 || ^20.3.0 || >=21.0.0","- Consult the installation documentation:","    See https://sharp.pixelplumbing.com/install","    at Object.<anonymous> (/var/task/node_modules/sharp/lib/sharp.js:114:9)","    at Module._compile (node:internal/modules/cjs/loader:1198:14)","    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)","    at Module.load (node:internal/modules/cjs/loader:1076:32)","    at Function.Module._load (node:internal/modules/cjs/loader:911:12)","    at Module.require (node:internal/modules/cjs/loader:1100:19)","    at require (node:internal/modules/cjs/helpers:119:18)","    at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:10:1)","    at Module._compile (node:internal/modules/cjs/loader:1198:14)","    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)"]}

After I upgrade runtime: lambda.Runtime.NODEJS_16_X, to runtime:lambda.Runtime.NODEJS_16_X,  I got the following error.  And NODEJS_20 seems not defined by cdk. likely due to this issue not closed? https://github.com/aws/aws-cdk/issues/27003

2024-01-15T09:24:43.647Z	undefined	ERROR	Uncaught Exception 	{"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'aws-sdk'
Require stack:
- /var/task/index.js
- /var/runtime/index.mjs","stack":["Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'","Require stack:","- /var/task/index.js","- /var/runtime/index.mjs","    at _loadUserApp (file:///var/runtime/index.mjs:1087:17)","    at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)","    at async start (file:///var/runtime/index.mjs:1282:23)","    at async file:///var/runtime/index.mjs:1288:1"]}

I also tried to use node-16 locally, but it does not work with cdk,

Is it possible for the repo to get an upgrade of major depedencies?

nvm. I noticed the latest main has upgrade to node-20


fix: events/codedeploy.go:16:2: SA9004 (staticcheck)

Issue #, if available:
It fixes the following staticcheck warning.
events/codedeploy.go:16:2: SA9004: only the first constant in this group has an explicit type (staticcheck)
	CodeDeployDeploymentStateFailure CodeDeployDeploymentState = "FAILURE"
	^

The warning is shown because CodeDeployDeploymentState constants have unexpected type.
package main

import (
	"fmt"

	"github.com/aws/aws-lambda-go/events"
)

func main() {
	fmt.Printf("%10[1]s: %[1]T\n", events.CodeDeployDeploymentStateFailure)
	fmt.Printf("%10[1]s: %[1]T\n", events.CodeDeployDeploymentStateReady)
	fmt.Printf("%10[1]s: %[1]T\n", events.CodeDeployDeploymentStateStart)
	fmt.Printf("%10[1]s: %[1]T\n", events.CodeDeployDeploymentStateStop)
	fmt.Printf("%10[1]s: %[1]T\n", events.CodeDeployDeploymentStateSuccess)
}

got:
   FAILURE: events.CodeDeployDeploymentState
     READY: string
     START: string
      STOP: string
   SUCCESS: string

want:
   FAILURE: events.CodeDeployDeploymentState
     READY: events.CodeDeployDeploymentState
     START: events.CodeDeployDeploymentState
      STOP: events.CodeDeployDeploymentState
   SUCCESS: events.CodeDeployDeploymentState

Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Codecov Report

Merging #322 into master will decrease coverage by 1.82%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #322      +/-   ##
==========================================
- Coverage   74.46%   72.63%   -1.83%
==========================================
  Files          18       18
  Lines         834      720     -114
==========================================
- Hits          621      523      -98
+ Misses        150      134      -16
  Partials       63       63

Impacted Files
Coverage Δ

lambda/rpc.go
55.55% <0.00%> (-8.09%)
:arrow_down:

events/epoch_time.go
77.77% <0.00%> (-4.05%)
:arrow_down:

events/cloudwatch_logs.go
60.00% <0.00%> (-3.64%)
:arrow_down:

lambda/runtime_api_client.go
75.00% <0.00%> (-2.78%)
:arrow_down:

lambda/invoke_loop.go
72.00% <0.00%> (-2.55%)
:arrow_down:

events/attributevalue.go
80.32% <0.00%> (-2.44%)
:arrow_down:

cmd/build-lambda-zip/main.go
33.82% <0.00%> (-2.29%)
:arrow_down:

lambda/panic.go
87.87% <0.00%> (-2.13%)
:arrow_down:

events/duration.go
42.85% <0.00%> (-1.59%)
:arrow_down:

lambda/function.go
86.84% <0.00%> (-1.53%)
:arrow_down:

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 55dc88b...285d478. Read the comment docs.


App Keygen

Coordinator Outbox
Current approach for handing coordinator outbox is to add a Arc<Mutex<VecDeque<..>> to the FfiCoordinator pending_for_outbox.

In the message handling loop we read messages into recv_device_message.
CoordinatorSend responses extend pending_for_outbox
At the end of the loop for this thread we drain all the messages, sending them to the appropriate recipient (ToDevice)

Outside of this loop, we can also add messages topending_for_outbox with a function like coordinator.do_keygen(). Which the other message thread above will process.
edit: there's also this branch app-sign as a v early demo of single device signing

Working in Android but not in Linux

Workflow:

The Generate Key button is hidden until at least one device is named.

The threshold slider is hidden until there are at least two named devices.

After finishing keygen, the app will alert prompt the user to confirm what they see matches the devices, the user response acks/nacks keygen on the coordinator.

If acked, the user is then taken to a screen which shows the public key (will later become a home-screen for the wallet).

Not addressed in this PR:

MaterialApp routes (to get to a permanent KeyDisplayPage page), you'll see you can still click the back arrow.
Saving of coordinator state to flash.
The workflow of "Yes/No" between the devices and coordinator still does not interact (e.g. Clicking No on coordinator does not send Cancel to devices, or otherwise)

I'd like to squash the keygen related commits before merging (messy, forgot to add files)

I've made major additions to this PR. We now land on "key list" page (which for now only ever has one key). You can start generating a new key from there. This brings us to the threshold selection (in the future probably also device selection page). From there you start the actual work of generating and end up having to confirm on each device and the coordinator.
I didn't end up doing the final confirmation message from coordinator to devices since the PR was big enough and I think you get the idea.
I've figured out a bit about how widget positioning, sizing and how the Navigator and learned a lot about designing flutter things.
One thing I also noticed is that the app takes a lot of CPU on linux. This seems to be because of our device polling loop. I'm surprised at this but haven't tried to debug it.

tested ACK

One thing I also noticed is that the app takes a lot of CPU on linux. This seems to be because of our device polling loop. I'm surprised at this but haven't tried to debug it.

I'm seeing 20% CPU usage

Reviewed rust code changes and looked over most of the dart, only minor questions above. Thanks for fixing the orientation and getting rid of those buggy Extends(..).
Tested on linux and it's awesome! Only suggestions (can wait for a later PR) is to bring back the old device list style(?), and make the "Press if all devices show..." a separate button appear to underneath the list.
Did we decide that we would always show the threshold slider on keygen page? Even when there are no devices or for 1-of-1?

I just added the device list to the name confirmation screen. The idea behind it is to always show all the devices when you are asking the user to do something with one of them.

Reviewed rust code changes and looked over most of the dart, only minor questions above. Thanks for fixing the orientation and getting rid of those buggy Extends(..).

What's Extends?

Tested on linux and it's awesome! Only suggestions (can wait for a later PR) is to bring back the old device list style(?), and make the "Press if all devices show..." a separate button appear to underneath the list.

I put the color back to how it was.

Did we decide that we would always show the threshold slider on keygen page? Even when there are no devices or for 1-of-1?

There are no decisions being made here. At the moment we are just trying to learn. I have made some interesting UI elements that maybe aren't strictly necessary but again this is just me trying to learn how this whole system works. The most important thing is to get every workflow step that needs to be there present in the UI and to make the workflows robust. There's still a few things missing here even from keygen and naming.
So I guess try to avoid making UI tweaks. Do radical changes. Do experiments that we can learn something from. I predict almost nothing we do right now will exist in 6 months time so there is really no useful gains to be had from small improvements yet.
Of course if something is really bothering you because and detracts from a demo feel etc just go ahead and change it in another PR (no need to ask permission or discuss).


blank screenshots for pages with meta refesh

getting blank screenshots with pages which have meta refresh
what is the best way to handle meta refresh
let page = await browser.newPage();
        await  page.setUserAgent('Mozilla/5.0 (Windows NT x.y; rv:10.0) Gecko/20100101 Firefox/10.0');
        page.setViewport({width: options.width, height: options.height});
        await page.goto(srcUrl, {timeout: self.timeout, waitUntil: 'networkidle',
          networkIdleTimeout: 2000});

await page.screenshot({
          path: '/tmp/test.png',
          clip: {x: 0, y: 0, width: options.width, height: options.height}
        });

sample page hxxp://www.snehpubliclibrary.org/my.adp/adp
warning: redirects to phishing site
sample code
<html xmlns='http://www.w3.org/1999/xhtml'>^M
<head>
<title></title>
<meta http-equiv='refresh' content='0;URL= login.php'>
</head>
</html>

That's because the interstitial has been shown.
We might want to do something about it.

I am see this issue with other kind of redirects as well. There is no delay on this page. chrome resolves it right away
For example:
warning this is a phishhing site
hxxp://apple-itunes-inc.com/?DYYHQ=36981.html

For example:
warning this is a phishing site
hxxp://apple-itunes-inc.com/?DYYHQ=36981.html

This throws me into interstitial as well.

The following worked for me:
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({headless: false});
  let page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Windows NT x.y; rv:10.0) Gecko/20100101 Firefox/10.0');
  page.goto('http://www.snehpubliclibrary.org/my.adp/adp', {
    waitUntil: 'networkidle'
  });
  page.on('framenavigated', doScreenshot);

  async function doScreenshot(frame) {
    if (frame.url() !== 'http://www.snehpubliclibrary.org/my.adp/adp/login.php')
      return;
    console.log('waiting for navigation');
    await page.waitForNavigation({waitUntil: 'networkidle'});
    console.log('screenshot!');
    await page.screenshot({
      path: 'test.png',
    })
  }
})();


OpenAI output null values for keys are converted to True in validated response

Describe the bug
OpenAI output null values for keys are converted to True in the validated response
To Reproduce
Steps to reproduce the behavior:
from typing import List, Dict, Optional, Any, Union

import openai
from pydantic import BaseModel as BaseModel, ConfigDict, Field
import sys

from guardrails.validators import ValidChoices

if sys.version_info >= (3, 8):
    from typing import Literal
else:
    from typing_extensions import Literal

metamodel_version = "None"
version = "1.0"

class ConfiguredBaseModel(BaseModel):
    model_config = ConfigDict(
    validate_assignment=True,
    validate_default=True,
    extra='forbid',
    arbitrary_types_allowed=True,
    use_enum_values = True)

class Person(ConfiguredBaseModel):
    """
    A human being regarded as an individual.
    """
    name: str = Field(..., description="""Name of the person""")
    birth_date: Optional[str] = Field(None, description="""date of birth of person""")
    death_date: Optional[str] = Field(None, description="""date of death of person""")
    salary: Optional[str] = Field(None, description="""salary of the given person""")
    occupation: Optional[str] = Field(None, description="""salary of the given person""", validators=[ValidChoices(choices=['Politician'], on_fail='noop')])

prompt = """Extract person information from the give text. Each extracted value should be valid.

text= "Elon Reeve Musk (born June 28, 1971) is a businessman and investor. Musk is the founder, chairman, CEO and chief technology officer of SpaceX; angel investor, CEO, product architect and former chairman of Tesla, Inc.; owner, chairman and CTO of X Corp.; founder of the Boring Company and xAI; co-founder of Neuralink and OpenAI; and president of the Musk Foundation. He is the wealthiest person in the world, with an estimated net worth of US$219 billion as of November 2023, according to the Bloomberg Billionaires Index, and $241 billion according to Forbes, primarily from his ownership stakes in Tesla and SpaceX."
${gr.complete_json_suffix}"""

import guardrails as gd

from rich import print

guard = gd.Guard.from_pydantic(output_class=Person, prompt=prompt)

response = guard(
    openai.chat.completions.create, model="gpt-3.5-turbo", #"gpt-4-1106-preview",
    prompt= guard.base_prompt
)

print(response)

Expected behavior
A clear and concise description of what you expected to happen.
Library version:
Version (e.g. 0.1.5)
Additional context
Add any other context about the problem here.

Hi, thanks for reporting this issue, we'll look into it.

My hunch here is that null is being read in as a Truthy string, and then getting translated to True. They should instead be dropped if the field is optional. if the field is mandatory, we should treat it as an error in skeletal validation.

@irgolic , I'm following the progress.

The offender appears to be here:
https://github.com/guardrails-ai/guardrails/blob/02ba65cd9a49e7bdbd5ecc700b0ea97d5cc551be/guardrails/utils/json_utils.py#L40C9-L41C24
By commenting out these lines, I am able to avoid this issue. That being said, I'm not totally sure what's going on here. @irgolic any ideas what this is doing?

Hey all, any updates here?

Hello @LakshmiPanguluri, we support multiple json suffixes to be added to the prompt; those can be found here.
Replacing the current ${gr.complete_json_suffix} with ${gr.complete_json_suffix_v2} works better in this case, and does not output null values in the first place. If the information is not given in the prompt itself and if the field is optional, the LLM won't output the field and it's value.
Also, I noticed a few typos in the prompt. Here's a corrected prompt (which also includes better designing according to multiple prompt designing guides available online):
prompt = """Extract person information from the given text. Each extracted value should be valid.

text:
Elon Reeve Musk (born June 28, 1971) is a businessman and investor. Musk is the founder, chairman, CEO and chief technology officer of SpaceX; angel investor, CEO, product architect and former chairman of Tesla, Inc.; owner, chairman and CTO of X Corp.; founder of the Boring Company and xAI; co-founder of Neuralink and OpenAI; and president of the Musk Foundation. He is the wealthiest person in the world, with an estimated net worth of US$219 billion as of November 2023, according to the Bloomberg Billionaires Index, and $241 billion according to Forbes, primarily from his ownership stakes in Tesla and SpaceX.

${gr.complete_json_suffix_v2}
"""

@tbrownio Great catch!
Now as far as why we convert the optional fields' values to True if null, needs some more research. Let us get back to you on this as soon as we can.

@tbrownio We looked into this in a bit more detail. When the field is optional and the value is null, we want to exit earlier and hence we return True from the base class - Placeholder's verify method. Once we get that in e.g here and multiple other places, instead of returning the super_result directly we should return None instead so as to avoid the main issue here. We're planning to add this fix soon.

A fix has been added for this issue! Merged in main, for now try using the latest code from main, until it makes it way into the package PyPI.
Closing this issue. @LakshmiPanguluri @tbrownio


Add Fuse network support

Expected behavior
Being able to use Fuse in the supported networks
Actual behavior
Fuse is not a supported network yet
Steps to reproduce the behavior
Browse https://snapshot.page/#/networks

should be closed by #227


Support for prometheus

Hi there,
it would be great to get the fabio metrics also to a prometheus instance.
Is there any chance for ?
greetz
Equilibri0

Somehow I knew that this was going to come :) I'm working on another metrics ticket right now and thought the same thing. I'll add it to the list.

+1

+1

+1

+1 for me too !

+1 that'd be cool!

Any news on the Prometheus /metrics endpoint ?
This feature would be very helpful to us to measure things like requests throughput, call latencies and status codes, globally and maybe per entrypoint/endpoint as well ? :)

I'd love to see prom support, but in the meantime you should be able to work-around it with statsd in fabio and https://github.com/prometheus/statsd_exporter

I need to wrap my head around on how to configure the naming and to expose the tags. The issue is that some metrics system want a single string as name which flattens all parts of the name into a single value. Others have a base name and other parts as a tag. What makes this a bit more complex is that some of the names are auto-generated and the pattern is configurable. So either this mechanism needs to work for all providers or I need to come up with something else. It probably isn't that difficult but that's the hold-up. Same for #165 .

Any progress on updating the internal Registry api to support labels? It would be a boon to both prometheus support as well as datadog style statsd+labels and others.
IMHO, part of the solution could be as simple as passing into the Registry's update functions a context object which contains the same information that is currently being used to flatten out into a name. And leave it up to the Registry to do the work of either flattening in accordance to the configured template OR using the attributes of the context in whatever other manner makes sense.

+1

work on consul 1.0 has kept me quite busy. I plan to go through open issues soon.

+1

working on it.

@magiconair I'd love to use this feature natively instad of the statsd prometheus exporter. Do you need a hand with this?. Do you have a branch with this feature where I can collaborate?.
On a different subject Hashicorp's Nomad uses their on go-metrics  (https://github.com/armon/go-metrics) package that translates metrics directly to graphite, statsd and prometheus. I haven't evaluated how difficult would be to change the current go-metrics package for his one, but seems like this would solve a lot of issues and remove some extra dependencies as well.

@marcosnils I'd love to have some help here to push this forward. I have an idea on how to do this since the problem is a bit bigger than just supporting prometheus. Look for the issues tagged theme-metrics. In short, none of the existing metrics libs solve all the problems and there needs to be some fabio specific glue code. My current thinking is to write a layer which handles the naming issues and configuration and then uses go-kit/metrics drivers since they are complete and seem to be maintained. Shall we do a conf call to discuss the approach and then document it here? Where are you located?

If someone else wants to help with this, too then pls just shout out.

@magiconair @marcosnils I'd like to help out with this if it's still needed.

@magiconair has there been any progress on labeled metrics? I'm also willing to help with this.

I've finally got something that I think is worth working on. Please see #476 in the metrics4 branch. Right now I'm trying to validate the approach. So if someone wants to write a proof-of-concept provider for prometheus using the go-kit/metrics driver then this would be awesome. This would then allow us to integrate all existing go-kit/metrics drivers which covers most of the open issues.
Lets keep the discussion on the metrics in #476 since other backends are also affected.
I'm not sure how we can collaborate on a PR on Github. I assume you fork the branch and create a patch. I'll try to incorporate patches into the metrics4 branch so that we all have the same base.
Thanks for your patience. This has been bugging me for a while so lets solve this properly.

I'm not sure how we can collaborate on a PR on Github

We can fork your code and open a PR to the metrics4 branch directly. If you merge your PR will get updated automatically.
Thanks for the hard work @magiconair . I'll give it a look this week and contribute in whatever I can.

@marcosnils did you have any luck diving it?

Not really, been extremely busy with some other stuff. Might be able to take a look some time this week. Can't promise anything though

Any updates on this?

What is the current recommended method for getting stats from fabio to prometheus?

@marcosnils Do you have any answers?

@KHiis not really. I'm currently using fabio statsd support and  statsd_exporter (https://github.com/prometheus/statsd_exporter) to push metrics to prom. Would love to add native prom support it's just that I don't have the time.

See https://github.com/fabiolb/fabio/pull/476#issuecomment-434048436

@marcosnils can you provide statsd_exporter mappings? Can't figure out why my mappings don't working.

@kikdevops here's my mapping config
mappings:
- match: ^([^.]*)\.([^.]*)--http.status.([^.]*).count
  match_type: "regex"
  name: "fabio_http_status_count"
  labels:
    code: "$3"
    instance: "$1"
- match: .
  match_type: "regex"
  action: drop
  name: dropped

@marcosnils Thanks a lot.

Does anyone have a full mapping config? I've tried @marcosnils's config and I don't see any fabio metrics on statsd exporter. I don't see any errors in the debug logs of exporter or Fabio.


I see no way to turn it on and off, short of disabling the extension

Other night mode type add-ons come with an off switch.  I don't see one for this, though that could possibly be due to the fact that the user interface is far too small for me to read #360
It seems all I can do is disable the add-on in Firefox's "Add-ons and themes" tab.

If the UI is too small for you then you can use keyboard shortcuts
Alt+Shift+L - toggle current website
Alt+Shift+M - toggle extension


How to use OCR on a specified region on the screen instead of using read step?

from user query

Following block of code works. Details on Region parameters - http://doc.sikuli.org/region.html#creating-a-region-setting-and-getting-attributes
Details on passing info back to TagUI - https://github.com/kelaberetiv/TagUI/issues/149
vision ocr_text = Region(760, 460, 440, 60).text()
vision tagui_text = open('tagui.sikuli/tagui_sikuli.txt','w')
vision tagui_text.write(ocr_text); tagui_text.close()
vision_result = fetch_sikuli_text()
echo vision_result

In practice we can't really provide a pre-captured screenshot as input to find a match to perform OCR. Because the textual contents will be different and the pre-captured image would not match. Best approach probably is by specifying the coordinates as above. But this only works if the window or region of interest always appears at the same place.

sharing reply from user, on his method base on anchoring to an image -
vision begin

# find the anchor element
tagui_text = open('tagui.sikuli/tagui_sikuli.txt','w')
reg = find('full_path/pdf_anchor.png')

# offset from the anchor element and perform OCR
ocr_text = Region(reg.x + 130, reg.y + 68, 1116, 683).text()
tagui_text.write(ocr_text)
tagui_text.close()

vision finish

vision_result = fetch_sikuli_text()
echo "result = " + vision_result


[Bug]: Docker Installation problem

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch name
main
Commit ID
...
Other environment information
Docker version 24.0.5
vm.max_map_count = 262144
WSL

Actual behavior
When I execute:  docker logs -f ragflow-server
I get
stan@PLWA1189NA:~/ragflow/ragflow/docker$ docker logs -f ragflow-server
[HUQIE]:Build default trie
[HUQIE]:Build default trie
[HUQIE]:Build trie /ragflow/rag/res/huqie.txt
[HUQIE]:Build trie /ragflow/rag/res/huqie.txt
[HUQIE]:Build default trie
[HUQIE]:Build trie /ragflow/rag/res/huqie.txt
[WARNING] [2024-04-11 20:50:02,717] [synonym.__init__] [line:24]: Realtime synonym is disabled, since no redis connection.
[WARNING] [2024-04-11 20:50:02,786] [synonym.__init__] [line:24]: Realtime synonym is disabled, since no redis connection.
[WARNING] [2024-04-11 20:50:03,021] [synonym.__init__] [line:24]: Realtime synonym is disabled, since no redis connection.
[WARNING] Load term.freq FAIL!
Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/utils/hub.py", line 398, in cached_file
    resolved_file = hf_hub_download(
                    ^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/ragflow/rag/res/bge-large-zh-v1.5'. Use `repo_type` argument if needed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ragflow/rag/llm/embedding_model.py", line 30, in <module>
    flag_model = FlagModel(os.path.join(
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/FlagEmbedding/flag_models.py", line 19, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 767, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 600, in get_tokenizer_config
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/utils/hub.py", line 462, in cached_file
    raise EnvironmentError(
OSError: Incorrect path_or_model_id: '/ragflow/rag/res/bge-large-zh-v1.5'. Please provide either the path to a local folder or the repo_id of a model on the Hub.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/root/miniconda3/envs/py11/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='hf-mirror.com', port=443): Max retries exceeded with url: /BAAI/bge-large-zh-v1.5/resolve/main/tokenizer_config.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)')))

Expected behavior
No response
Steps to reproduce
Above

Additional information
No response

+1

I have something like this. I can get all components of the docker to return healthy, I have vm.max_map_count = 262144. But it returns 404 in the browser.

I have something like this. I can get all components of the docker to return healthy, I have vm.max_map_count = 262144. But it returns 404 in the browser.

@cornpo for it to return you a working containers did you had to preconfigure them in specific way?

No, I didn't change the .env
My first problem was the "container unhealthy" problem due to wrong docker-compose version. I upgraded to requirements. Then "docker-compose up -d." Checked docker log and they all said healthy. But it would 404 in the browser for the page even though Werkzeug was serving.

No, I didn't change the .env
My first problem was the "container unhealthy" problem due to wrong docker-compose version. I upgraded to requirements. Then "docker-compose up -d." Checked docker log and they all said healthy. But it would 404 in the browser for the page even though Werkzeug was serving.

Remove the port you enter in browser.

Is there an existing issue for the same bug?

[x]  I have checked the existing issues.

Branch name
main
Commit ID
...
Other environment information
Docker version 24.0.5
vm.max_map_count = 262144
WSL

Actual behavior
When I execute: docker logs -f ragflow-server
I get
stan@PLWA1189NA:~/ragflow/ragflow/docker$ docker logs -f ragflow-server
[HUQIE]:Build default trie
[HUQIE]:Build default trie
[HUQIE]:Build trie /ragflow/rag/res/huqie.txt
[HUQIE]:Build trie /ragflow/rag/res/huqie.txt
[HUQIE]:Build default trie
[HUQIE]:Build trie /ragflow/rag/res/huqie.txt
[WARNING] [2024-04-11 20:50:02,717] [synonym.__init__] [line:24]: Realtime synonym is disabled, since no redis connection.
[WARNING] [2024-04-11 20:50:02,786] [synonym.__init__] [line:24]: Realtime synonym is disabled, since no redis connection.
[WARNING] [2024-04-11 20:50:03,021] [synonym.__init__] [line:24]: Realtime synonym is disabled, since no redis connection.
[WARNING] Load term.freq FAIL!
Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/utils/hub.py", line 398, in cached_file
    resolved_file = hf_hub_download(
                    ^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/ragflow/rag/res/bge-large-zh-v1.5'. Use `repo_type` argument if needed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ragflow/rag/llm/embedding_model.py", line 30, in <module>
    flag_model = FlagModel(os.path.join(
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/FlagEmbedding/flag_models.py", line 19, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 767, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 600, in get_tokenizer_config
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/utils/hub.py", line 462, in cached_file
    raise EnvironmentError(
OSError: Incorrect path_or_model_id: '/ragflow/rag/res/bge-large-zh-v1.5'. Please provide either the path to a local folder or the repo_id of a model on the Hub.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/root/miniconda3/envs/py11/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='hf-mirror.com', port=443): Max retries exceeded with url: /BAAI/bge-large-zh-v1.5/resolve/main/tokenizer_config.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)')))

Expected behavior
No response
Steps to reproduce
Above

Additional information
No response

If you do not have the internet connection, or are unable to ping hf-mirror.com, I suggest you manually download the resource from InfiniFlow/deepdoc into local folder, say, ~/deepdoc. Then, change the docker-compose.yml, add a volumes, like - ~/deepdoc:/ragflow/rag/res/deepdoc

@KevinHuSh
Can you please provide an example of such changes?
Why does the installation depend on the hf-mirror?

去这里手动下载
https://huggingface.co/InfiniFlow/deepdoc/tree/main

Ok I found the solution. Go to the docker-compose.yml file.
There you will find this code
    environment:
      - TZ=${TIMEZONE}
      - HF_ENDPOINT=https://huggingface.com

You need to change it to this ->
    environment:
      - TZ=${TIMEZONE}
      - HF_ENDPOINT=https://hf-mirror.com


fix: update deprecated EOS.undent block in formula

Reference: Homebrew/brew#3738
Thanks for add formula for Homebrew. I fixed error in installation process.

@Hazealign Thank you.


Support for vSphere 8

Feature Request
Detailed Description
bosh-vsphere-cpi currently supports vSphere 6.5/6.7/7.0 but vSphere 8 ist GA for nearly 6 Month now. Support for vSphere 8 would be needed.
Context
We are using Tnazu Kuberntes Grid Integrated Edition which only supports vSphere 7. One reason seems to be that bosh-vsphere-cpi does not support vSphere 8 yet.
In the near future other customers need to upgrade there vSphere environments to Version 8.
Possible Implementation
Sorry, but i don't now anything about the code base of bosh-vsphere-cpi.
Complexity

[x ] Medium - Change requires some thought, but is relatively isolated
As i mention above, i don't now anything about your code base but i would think that most of the functionality of vSphere 7 will be the same for Version 8.

@JoschuaA4 :

bosh-vsphere-cpi currently supports vSphere 6.5/6.7/7.0 but vSphere 8 ist GA for nearly 6 Month now. Support for vSphere 8 would be needed.

Absolutely! We want to certify the BOSH vSphere CPI with vSphere 8. In fact, we're actively working on it. The big challenge we're currently overcoming is setting up the testbeds to certify vSphere 8. To go a step further, the portion that's proving the most challenging is making sure we test against NSX 4.0.1 (the only version of NSX that supports vSphere 8).
From an informal standpoint, the BOSH vSphere CPI works fine out-of-the-box with vCenter 8—my personal foundation has been running on vCenter 8 for the last few months (unfortunately no NSX). But that's not enough to certify vSphere 8 compatibility. We want to make sure every change we make to the CPI code is subjected to a battery of tests against vSphere 8 + NSX 4.0.1 environments, and we're working on that right now.

v90 is now tested against vSphere 8. I don't believe there were any actual changes in the CPI to work with vSphere 8, so older versions are also likely to work, but now we test against it.


ModuleNotFoundError: No module named 'Plugins'

On Debian 11 to install, chromium-browser is replaced with chromium so the installation is slightly modified like this
pip3 install bs4 pyppeteer
apt -y install chromium

Installing everything, it doesn't run and gives me the following error message
Traceback (most recent call last):
  File "/root/snake.py", line 2, in <module>
    from Plugins.base import Base
ModuleNotFoundError: No module named 'Plugins'

If you run Snake-Ping outside of the actual directory yea.
python3 Snake-Ping/snake.py
Run it within the directory and it should work fine.


style[cartesian]: readability improvements and more type hints

Description
This PR detaches a couple of cleanups in the dace backend from the in-progress gt4py/dace bridge: mostly readability improvements and some easy type hints. There's also the occasional unused variable / argument in here.
Requirements

[ ] All fixes and/or new features come with corresponding tests.
N/A: mypy is happy with the new type hints
[ ] Important design decisions have been documented in the approriate ADR inside the docs/development/ADRs/ folder.
N/A

/cc @twicki @FlorianDeconinck I've been reading more code "outside my comfort zone" today leading to this small cleanup PR

ping @FlorianDeconinck / @twicki


Use upstream version of tokio-tar

tokio-tar 0.3.1 got released, including all changes from the fork currently used, switch over to that one.
The https://github.com/neondatabase/tokio-tar.git fork can be removed now too.

nice to meet you, @SomeoneToIgnore  :)

Ah, the magic for external contributors  doesn't work yet..
Run gh pr --repo "${GITHUB_REPOSITORY}" edit "${PR_NUMBER}" --remove-label "approved-for-ci-run"
GraphQL: Resource not accessible by integration (removeLabelsFromLabelable)

And manually created ci-run/pr-4722 hasn't been picked up by CI. I guess a rebase should help with this part (I merged my changes after this PR was created).
@SomeoneToIgnore would you mind rebasing your branch on the latest main?
It's nice to see you here again, Kirill ❤

Glad to see external contributors' lives improving, rebased & ready for the next try.

Notes for myself
One more thing that doesn't work as I expected initially, CI runs for different triggers (pull_request for PRs and push for branches) are reported as different checks:

A possible workaround — create a PR from a ci-run/pr-* branch (and do not trigger CI on branches)
gh pr create --draft --head ci-run/pr-4722 --title '[DO NOT MERGE] CI run for #4722' --body='#4722'

Also we probably want to avoid triggering anything on third-party CI as is (because it'll fail).
And it'll be nice to add a GitHub autocoment with test results to "a parent" PR.

Does @bayandin want to test this automation more or should we just merge it?


Uncaught TypeError: Cannot read property 'classList' of null

calling of Pane.splitRight()/Pane.splitLeft()/Pane.splitDown()/Pane.splitUp().
can not provide more detailed reproduction sequence, since issue appears rarely and randomly and too difficult to debug it. Only can say that issue appears on startup and pane fetched by atom.workspace.getActivePane()

Atom: 1.19.1 x64
Electron: 1.6.9
OS: Mac OS X 10.10.5
Thrown From: 'api-workbench' package v0.8.46
Stack Trace
Uncaught TypeError: Cannot read property 'classList' of null
At /Applications/Atom.app/Contents/Resources/app/src/pane-resize-handle-element.js:25

TypeError: Cannot read property 'classList' of null
    at HTMLElement.PaneResizeHandleElement.attachedCallback (/Applications/Atom.app/Contents/Resources/app/src/pane-resize-handle-element.js:25:45)
    at HTMLElement.PaneAxisElement.childRemoved (/Applications/Atom.app/Contents/Resources/app/src/pane-axis-element.js:111:21)
    at HTMLElement.PaneAxisElement.childReplaced (/Applications/Atom.app/Contents/Resources/app/src/pane-axis-element.js:122:12)
    at Function.module.exports.Emitter.simpleDispatch (/Applications/Atom.app/Contents/Resources/app/node_modules/event-kit/lib/emitter.js:25:20)
    at Emitter.module.exports.Emitter.emit (/Applications/Atom.app/Contents/Resources/app/node_modules/event-kit/lib/emitter.js:141:34)
    at PaneAxis.module.exports.PaneAxis.replaceChild (/Applications/Atom.app/Contents/Resources/app/src/pane-axis.js:209:33)
    at Pane.module.exports.Pane.split (/Applications/Atom.app/Contents/Resources/app/src/pane.js:984:27)
    at Pane.module.exports.Pane.splitDown (/Applications/Atom.app/Contents/Resources/app/src/pane.js:972:25)

Non-Core Packages
api-workbench 0.8.46
atom-editor-logger 0.0.4
busy-signal 1.3.0
file-icons 1.7.14
intentions 1.1.2
linter 2.1.4
linter-ui-default 1.5.4

Thanks for the report!  It looks like this was also reported in api-workbench 1 and you closed the issue from https://github.com/mulesoft/api-workbench/commit/0178038a6c7abaf09526f70fe0532bcda01369a2?
Just to clarify, do you think there's still a bug on the Aton side?

@rsese, yes i thing there is still Atom's bug, my fix is just workaround until u fix it on your side(i hope), of course may be we doing something wrong with Panes, but i can't see how, currently

I can reproduce this rarely using the steps in https://github.com/mulesoft/api-workbench/issues/511. I did the steps 10 times and saw the exception once.
/cc: @as-cii @nathansobo Can this be an editor regression? From what I see the latest version of api-workbench was published on June 14th https://github.com/mulesoft/api-workbench/commit/a28459a954bce38c024a4a875c560bbbbe0a6d29 and it was not until Atom 1.19 was in stable that the first report appeared.
/cc: @ungb Can you look at this and see if you can narrow down the repro steps?

@Ben3eeE This doesn't look related to the editor at all, but it could be related to the Electron upgrade. The timings of custom element lifecycle callbacks may have changed slightly. If you look at the stack it looks like we're firing an attachedCallback due to the removal of a child. It doesn't make much sense to me. It also doesn't make sense that an attachedCallback would be fired when an element isn't actually attached (as in, has no parent element). I wonder if we could just null guard this situation and move on.


will you add "search by txid " function?

yes, not sure when but will be done by querying the es-wrapper.


Add stacking tools

I've added a spectral stacking tool - stacking in spatial dimensions to create a stacked spectrum - and will soon add a spectral stacking tool also.
Question - where should these live?  @adrn, I've put them in analysis_utilities (...not to be abbreviated), which I think is a descriptive location, but what name would you recommend here?
@e-koch, @low-sky - you might have useful suggestions on naming / testing / other things to add to this?

Coverage decreased (-0.3%) to 75.244% when pulling 54e534dbf2d2a8a5ca3a72fe19377213f3765155 on keflavich:analysis_utils into 54e638099e85fe07117c92f3ef71c82e3898fc29 on radio-astro-tools:master.

I have an implementation for the shifting hidden away here. That repo has become the place for tools that I didn't seem to fit into existing packages (and a lot is experimental). You're welcome to take whichever of that code will work here (and I'm happy to help, too).
There's a couple of basic tests that can be adapted to work here, too.
I prefer doing the shifting in the Fourier plane since it minimizes the effect of finite channel sizes and the need to interpolate onto finer grids.

Thanks.  Fourier shifting is nice, but it's probably no better or worse than direct interpolation when you're working within the same cube.
  Want to try to merge it in with this?  Possibly replace what I did and bring in your tests & all?

I strongly advocate Fourier shifting since it preserves the noise properties of the spectrum for sub-channel shifts.

Sure. I'll clean/refactor somewhat before adding to the PR.

Presumably it preserves the noise properties because the spectrum was already created with an FT?  Is there any reason to believe direct interpolation does not?
(I have no problem using the FT-based shift, just trying to understand)

Coverage decreased (-0.6%) to 74.993% when pulling f75b57211e97697fbf3e1e560219739e40b4bb0c on keflavich:analysis_utils into 54e638099e85fe07117c92f3ef71c82e3898fc29 on radio-astro-tools:master.

Direct interpolation can be done right, but it's essentially Fourier interpolation (i.e., you have to interpolate over all the points anyways).  If you do something like a naive linear interpolation of half a channel shift, you put your output data as the average of the two points on either side: x_new = (x_i + x_(i+1))/2, which basically smooths and your noise drops by sqrt(2).  For free.  As noted, you can do better, but it seems like the complexity gains approach FTs.

@keflavich - I added a refactored version of my stacking functions and some simple tests.
I didn't touch stack_cube. I'm not sure if you meant for it to average between multiple lines, or shuffle and then average them.

Coverage increased (+0.1%) to 75.714% when pulling bc736c17a1a8fa0c9c7d7a6162619d2df66ff8e4 on keflavich:analysis_utils into 54e638099e85fe07117c92f3ef71c82e3898fc29 on radio-astro-tools:master.

 Seems to be unrelated to this PR

Coverage increased (+0.2%) to 75.82% when pulling 58f5b0c8d790fad361712d3c7b12c6148862c144 on keflavich:analysis_utils into 54e638099e85fe07117c92f3ef71c82e3898fc29 on radio-astro-tools:master.

Coverage increased (+0.2%) to 75.82% when pulling 58f5b0c8d790fad361712d3c7b12c6148862c144 on keflavich:analysis_utils into 54e638099e85fe07117c92f3ef71c82e3898fc29 on radio-astro-tools:master.

Coverage increased (+0.3%) to 75.931% when pulling b0cce013a06838b355bf952c279b3874bf795d7f on keflavich:analysis_utils into 54e638099e85fe07117c92f3ef71c82e3898fc29 on radio-astro-tools:master.

Looks good to me.  Do you want to add that last linear axis check or just call it done?

It's on line 208. vdiff_tol sets the maximum relative change the vdiff can have (default set to 0.01).


Lint

A long-overdue thanksgiving gift 🦃
Test URLs
Screenshot

Fix file-finder-buffer left margin

Before:

After:

I would not add actual changes in lint PRs because they get lost and there's no argument/screenshot for the change. Lint PRs are for refactorings and code cleanups, not bugfixes nor new features

because they get lost

I learned this the hard way :+1:

Let's not keep lint PRs around for too long. It's good as is for now. You can review the "restore" commits I made and reviews I left and open separate PRs for those.


22050Hz should be the baseline sample rate for synthesis quality

Here are the difference in spectrograms between sample rates as 16kHz and 22050Hz. Sadly I am afraid I have made a huge mistake. Now I have to re-sample all my 16KHz dataset to get better quality...

@begeekmyfriend  these two are trained with same datasets and with different sample rate？

效果听起来差距非常大吗？16k的合成听起来也还行就是没细节

Insignificant. Close it.


Class not found: my_project.NodeTree.001

If you make more than 1 NodeTree in a zblend .blend file then you will get this error on build:
"Haxe stderr: ..\Sources/Main.hx:10: characters 8-47 : Class not found : my_project.NodeTree.001"
"Haxe process exited with code 1"
Even if you do not use NodeTree.001 anywhere, it will still give that error. Also, renaming NodeTree.001 to something else like "NodeTree2" will still give the error message on build using the old name "NodeTree.001"

Good catch! Should be fixed now, the '.' was causing trouble in generated Haxe code.
'Clean' button now also removes compiled nodes, so if something goes wrong during node parsing cleaning may help get it back to working state.

Old project was still getting that error even after a clean so I started a new one with same node setup. No longer errors on build but if I make something in NodeTree.001 and assign that one to the object in the traits view then the cube doesn't render and I get these errors:
Firefox: "TypeError: cl is not a constructor kha.js:351:0"
Chrome: "kha.js:351 Uncaught TypeError: object is not a function"
"http://<IP_ADDRESS>:8000/favicon.ico Failed to load resource: the server responded with a status of 404 (File not found)"
You can go back to the first nodeTrue, make edits and use that one and it works fine though

Silly me, forgot to also take care of '.' character in nodes trait. Now it should really work(hopefully).
Let me know if you get any ideas on what to do next with nodes. Thinking about adding some kind of conditions. :)

It works thanks! I tried adding an 'Input' node to get mouse input but to no avail :( I started by extending the FloatNode like you did for SineNode and it worked with 1 value. But once I changed the variable name from 'f' to 'x' it stopped working. Adding in another variable 'y' also didn't work.
I've got my code up here if you have time to review it: https://drive.google.com/open?id=0Bwiz0Tss2NSqfkZESWhEQmpEcDFreGlSRk5fM3lYdVVJazdrWWRiQVphb1pNWmw5aUhVUjg&authuser=0
I'm a little new to Haxe but I'd love to contribute to ZBlend once I get it figured out :p


Add gateway claiming reference

Summary

Add gateway claiming doc; refs https://github.com/TheThingsIndustries/lorawan-stack/pull/2615
Screenshots

Changes

Add gateway claiming doc

Checklist

[x] Scope: The referenced issue is addressed, there are no unrelated changes.
[x] Run Locally: Verified that the docs build using make server, posted screenshots, verified external links. Test with HUGO_PARAMS_SEARCH_ENABLED=true if style changes will affect the search bar.
[x]  New Features Marked: Documentation for new features is marked using the new-in-version shortcode, according to the guidelines in CONTRIBUTING.
[x] Style Guidelines: Documentation obeys style guidelines in CONTRIBUTING.
[x] Commits: Commit messages follow guidelines in CONTRIBUTING, there are no fixup commits left.

@KrishnaIyer feel free to merge this


Fixing Dependabot/pip/django 2.2.10

The build has been failing because there are some models in the ratings tests(Beverage,Food,BeverageRating) have no migrations associated with them.  Because of this when you run the tests it attempts to synchronize the db and in doing so it attemots to create a FK to auth_user which does not exist.
By explicitly creating the migration this happens after the auth_user table is created.

Ive manually tested and all looks okay with django 2.2.10


fix build

old

new

Checked.
The wrong icon was replaced for the breadcrumb. Lucidehas removed the icon from their lib.


Patch h2 v0.4.0 for WasmEdge

This PR is based on h2 v0.4.0 and rebases the changes for usage with WasmEdge from the master branch.
Unfortunately, due to the rebase this is not fast-forward.
To avoid conflicts we could create a separate branch and tag for v0.4.0.
What do you think?

@0xE282B0 Thanks for the pull request!  Before we can merge this, we need you to sign off on each commit in the pull request.  Please add your sign-off to each commit using the --signoff option when making commits.  This indicates that you agree to the Developer Certificate of Origin (DCO).

Hello,
Most of the commits are rebased from the upstream project. But I can add a signoff 😊.

Rebased with signoff.

The 'client' and 'server' examples in the 'examples' folder are fine, but the 'akamai' example cannot compile successfully. This is because its dependency, 'tokio_rustls,' uses 'tokio' instead of 'tokio_wasi.'
Yes, there are many dependencies that need to be adjusted upwards, so we are currently working on finding a solution to this problem.


[BUG] Mirador from local file doesn't work with Aggregate Javascript

Posting this here as I'm not sure which repository to put it in. It might be a configuration issue with nginx in buildkit and/or the site-template.
I'm running the Isle Site Template with a Starter Site based on 1.0.0 (running Drupal 10.1).
I'm testing the Mirador PR that fixes the "use local library" option, so i'm on a page where an image is shown in mirador.
When javascript aggregation is off, Mirador and the image shows up from the local libraries folder (success!)
But when javascript aggregation is on (default), the Mirador viewer does not show up at all (a blank empty space is displayed where the mirador viewer would be).
The console shows (truncated a really long alphanumeric string for courtesy):
The resource from “https://islandora.dev/sites/default/files/js/js__KsmWjh3S9c-3z1l3qc3F9jG0StLevS-CTl-GnWUOlg.js?scope=footer&delta=2&language=en&theme=olivero&include=eJx9[...]” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
The network tab shows (same long string truncated):
GET | https://islandora.dev/sites/default/files/js/js__KsmWjh3S9c-3z1l3qc3F9jG0StLevS-CTl-GnWUOlg.js?scope=footer&delta=2&language=en&theme=olivero&include=eJx9[...] 500 Internal Server Error

Further info:
On Drupal 10 served by Apache on a bare metal Mac, this problem does not occur.
More further info:
Watchdog shows:
Error: Xdebug has detected a possible infinite loop, and aborted your script with a stack depth of '256' frames in Peast\Syntax\Parser->checkAsyncFunctionStart() (line 1405 of /var/www/drupal/vendor/mck89/peast/lib/Peast/Syntax/Parser.php). and "Location" was the really long URL truncated above.

This is very likely related to this Drupal issue which locates the problem between XDebug and Peast. Comment 14 points out the known issue on Peast's page whcih suggests raising the xdebug.max_nesting_level...


Visual Studio Code Cannot Detect Git in Source Control Tab

Type: Bug
Expected Result:
Few days ago, before i update my vscode to version 1.75.0 i have been able to used The Source Control icon in the Activity Bar on the left that will always indicate an overview of how many changes i currently have in my repository it showed the list of my repositories and it worked without problems.
Actual Result
After i update my vscode to version 1.75.0 i am not be able to use The Source Control icon in the Activity Bar on the left. It doesn't show lists of my repositories, it only show sentence, "The workspace currently open doesn't have any folders containing git repositories...."
Please help. This problem has been going on for days until now and hinders me from working
VS Code version: Code 1.75.0 (e2816fe719a4026ffa1ee0189dc89bdfdbafb164, 2023-02-01T15:23:45.584Z)
OS version: Windows_NT x64 10.0.22623
Modes:
Sandboxed: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz (8 x 2304)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
undefined

Memory (System)
7.80GB (0.53GB free)

Process Argv
--crash-reporter-id 98480a36-2e46-435f-97c9-fc66d46208b0

Screen Reader
no

VM
90%

Extensions (29)

Extension
Author (truncated)
Version

vscode-intelephense-client
bme
1.9.4

vscode-tailwindcss
bra
0.9.6

dscodegpt
Dan
1.0.12

vscode-eslint
dba
2.2.6

gitlens
eam
13.2.0

vs-code-xml-format
fab
0.1.5

copilot
Git
1.70.8099

hclappscancodesweep
HCL
1.2.2

beautify
Hoo
1.5.0

git-graph
mhu
1.30.0

fluent-icons
mig
0.0.18

isort
ms-
2022.8.0

python
ms-
2022.20.2

vscode-pylance
ms-
2023.1.30

jupyter
ms-
2022.11.1003412109

jupyter-keymap
ms-
1.0.0

jupyter-renderers
ms-
1.0.14

vscode-jupyter-cell-tags
ms-
0.1.6

vscode-jupyter-slideshow
ms-
0.1.5

cpptools
ms-
1.14.0

vscode-thunder-client
ran
2.3.4

music-time
sof
2.2.43

swdc-vscode
sof
2.6.42

sonarlint-vscode
Son
3.13.0

open-in-browser
tec
2.0.0

pdf
tom
1.2.2

vscode-icons
vsc
12.2.0

vscode-wakatime
Wak
23.2.0

material-theme
zhu
3.15.6

(1 theme extensions excluded)

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
azure-dev_surveyone:30548225
vsccc:30610678
pyindex848:30577860
nodejswelcome1:30587005
3biah626:30602489
89544117:30613380
pythonsymbol12cf:30657549
2i9eh265:30646982
vsccsb:30659829

I have the same problem after the update:). And I found that the reason is that there is one folder in my workspace not accessible. Then  I fix (or remove) the folder, and restart VSCode,  and VSCode can detect repos in my workspace as before.

/duplicate of #173226 and fixed in upcoming recovery release

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.
Happy Coding!


I have the table user already created

I want to implement the admin panel but I already have an users table this table doesn't have the remember toke, and instead of rol_id y have a column called tipo_usuario_id (it has the same goal but this database is in spanish) so I instead of roles table i have tipo_usuarios table, and I have a lot of different problems like that, so can I adapt it or it is just imposible?

@jiac0911 one sentence from Readme file: "Please note: QuickAdmin requires fresh Laravel installation and is not suitable for use on already existing project."


Deprecate knife cookbook create in favor of chef generate cookbook.

@chef/chef-maintainers

:+1: LGTM


Have unit tests for each action

Done in https://github.com/krzema12/github-actions-kotlin-dsl/commit/1570935b050ab4725ed48f3a74ec12506d6a27f8


Simplify Tap#remote_repo.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?

Extracted from https://github.com/Homebrew/brew/pull/16730 for easier review.

I'm not sure this is really simpler. It's a few less lines I guess.

Agreed with @apainintheneck, this is worse as it actually introduces more branching logic despite being fewer lines. Passing on this, sorry.

It's needed for Sorbet to work. Note the signature is added in a later commit in the original PR.

Feel free to suggest an alternative that only calls remote once.

Feel free to suggest an alternative that only calls remote once.

remote_repo = remote and then use that variable instead.

It's needed to work with the signature of Tap#remote, which is added in a later commit in the original PR.

In that case, I'm okay with whatever appeases the type system. That goes for other PR too.

@reitermarkus Maybe this shouldn't be a separate PR in that case though. It'd be easier to understand in context. That being said I appreciate you breaking up these in general PRs so they're easier to review.

Thanks @reitermarkus!


Developer Advice: Simulate multitouch with key or mouse

Hi, I'm a game developer and my QA teammate uses scrcpy to test our games in multiple remote phones with different specs. This is working nice so far, but we just have one issue: In this game (which is in a develop version) you can open the "cheat console" with a 3-finger touch, this is an important feature for the QA team because it helps them to quickly test their cases.
The PC doesn't have a touch screen (I read that multitouch is possible with it), so what we'd like is to assign some key or key-mouse combination to achieve this. I know this is a very specific feature, so I was thinking I can implement it myself locally (without making a PR) but first I want to know if that implementation is possible, and if it's possible I would like an advice on which files I would need to modify.
Thanks in advance.

To inject an event, call sc_controller_push_msh(). Here is an example to inject a touch event:
https://github.com/Genymobile/scrcpy/blob/c083a7cc9015b52e12b0fb78e5a488ab1f5b68f6/app/src/mouse_inject.c#L131-L144
You should probably call it from input_manager.c, keyboard_inject.c or mouse_inject.c.

Thanks! I'll give it a try. Do I have to send 3 touch events or is there a property in the event to configure the number of fingers?

3 successive sc_touch_events having different pointer_ids with action SC_TOUCH_ACTION_DOWN, then 3 successive sc_touch_events with action SC_TOUCH_ACTION_UP (having the same 3 pointer_ids as for the down action).

Hi again, I finished making the changes I needed but then had trouble trying to build. I'm trying to do the cross-compile with a Debian VM and according to the build instructions I don't need Android SDK if I use the prebuilt server, but the build was still failing and complaining about it. Then I installed Android studio and set the root sdk but I'm getting the same error.
`FAILURE: Build failed with an exception.

What went wrong:
Could not determine the dependencies of task ':server:compileDebugJavaWithJavac'.

SDK location not found. Define a valid SDK location with an ANDROID_HOME environment variable or by setting the sdk.dir path in your project's local properties file at '/home/friday/RiderProjects/scrcpy/local.properties'.
`

I think I'm doing something wrong, or do I need to build the server too because of the changes I made?

I think I'm doing something wrong, or do I need to build the server too because of the changes I made?
Its the same Error I had for multiple hours.
My setup: Windows PC with Debian Subsystem (basically installed like this: https://www.youtube.com/watch?v=JzAAxL3HrXU)
I used the build guide here: https://github.com/Genymobile/scrcpy/blob/master/doc/build.md
I had problems installing Java11, I ended up downloading it from here: https://jdk.java.net/archive/
and following this guide for installing Java: https://www.youtube.com/watch?v=72mz0-98vg8
The next error happened, with the missing android-sdk. My error happened with this command:
./release.sh

first, I realized it should be
sudo ./release.sh

instead. And then I realized it wanted the android-sdk, which the guide did not specify before. The guide mentioned android sdk platform tools, which is sadly not the same.
I followed the following guide to get android-sdk: https://www.youtube.com/watch?v=OGJOPnV2fDc
I got the android-sdk by installing android studio, going into the settings and copying the path from it, and putting it inside the local.properties file like this:
sdk.dir=/home/termii/Android/Sdk

Note that if your android sdk path starts with root/*, you need to reinstall android studio without sudo.

@Termiiii What are you trying to do? Is it related to this issue?

Windows PC with Debian Subsystem

( have never tested)

first, I realized it should be sudo ./release.sh

No, it should not. Do not execute that script as root.
If you just want to build the latest release, you can use the prebuilt server so you don't need the Android SDK at all.
Otherwise, yes, you need the Android SDK.

I tried to build client & server, because I want to modify the client later on. I am on Windows, so I assumed I need to follow the "Cross-compile from Linux"-section (assuming it is for Windows + Debian Subsystem)
Cross-compile from Linux
This is the preferred method (and the way the release is built).
From Debian, install mingw:
sudo apt install mingw-w64 mingw-w64-tools
You also need the JDK to build the server:
sudo apt install openjdk-11-jdk
Then generate the releases:
./release.sh
It will generate win32 and win64 releases into dist/.
If I don't execute the script as root, I get all kinds of errors:
termii@DESKTOP-PKFAPQQ:~/my$ cd scrcpy
termii@DESKTOP-PKFAPQQ:~/my/scrcpy$ ./release.sh
./gradlew clean
> Task :server:clean FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':server:clean'.
> java.io.IOException: Unable to delete directory '/home/termii/my/scrcpy/server/build'
    Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/release/com/genymobile/scrcpy/BuildConfig.java
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/release/com/genymobile/scrcpy
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/release/com/genymobile
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/release/com
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/release
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/debug/com/genymobile/scrcpy/BuildConfig.java
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/debug/com/genymobile/scrcpy
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/debug/com/genymobile
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/debug/com
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig/debug
    - /home/termii/my/scrcpy/server/build/generated/source/buildConfig
    - /home/termii/my/scrcpy/server/build/generated/source
    - /home/termii/my/scrcpy/server/build/generated/res/resValues/androidTest/debug
    - /home/termii/my/scrcpy/server/build/generated/res/resValues/androidTest
    - /home/termii/my/scrcpy/server/build/generated/res/resValues/release
    - /home/termii/my/scrcpy/server/build/generated/res/resValues/debug
    - and more ...

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s
2 actionable tasks: 1 executed, 1 up-to-date
make: *** [release.mk:50: clean] Error 1
termii@DESKTOP-PKFAPQQ:~/my/scrcpy$

I don't know if my root approach is working either, I am currently wondering what the current error is:
termii@DESKTOP-PKFAPQQ:~/my/scrcpy$ sudo ./release.sh
[sudo] password for termii:
./gradlew clean

BUILD SUCCESSFUL in 1s
2 actionable tasks: 1 executed, 1 up-to-date
rm -rf "dist" "build-test" "build-server" \
        "build-win32" "build-win64"
[ -d "build-test" ] || ( mkdir "build-test" && \
        meson setup "build-test" -Db_sanitize=address )
The Meson build system
Version: 1.0.1
Source dir: /home/termii/my/scrcpy
Build dir: /home/termii/my/scrcpy/build-test
Build type: native build
Project name: scrcpy
Project version: 2.1.1
C compiler for the host machine: cc (gcc 12.2.0 "cc (Debian 12.2.0-14) 12.2.0")
C linker for the host machine: cc ld.bfd 2.40
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (1.8.1)
Run-time dependency libavformat found: YES 59.27.100
Run-time dependency libavcodec found: YES 59.37.100
Run-time dependency libavutil found: YES 57.28.100
Run-time dependency libswresample found: YES 4.7.100
Run-time dependency sdl2 found: YES 2.26.5
Run-time dependency libavdevice found: YES 59.7.100
Run-time dependency libusb-1.0 found: YES 1.0.26
Checking for function "strdup" : YES
Checking for function "asprintf" : YES
Checking for function "vasprintf" : YES
Checking for function "nrand48" : YES
Checking for function "jrand48" : YES
Checking for function "reallocarray" : YES
Header "sys/socket.h" has symbol "SOCK_CLOEXEC" : YES
Configuring config.h using configuration
Program ./scripts/build-wrapper.sh found: YES (/home/termii/my/scrcpy/server/./scripts/build-wrapper.sh)
Build targets in project: 13

scrcpy 2.1.1

  User defined options
    b_sanitize: address

Found ninja-1.11.1 at /usr/bin/ninja
ninja -C "build-test"
ninja: Entering directory `build-test'
[88/114] Generating server/scrcpy-server with a custom command
(not invoking gradle, since we are root)
[114/114] Linking target app/scrcpy
./gradlew -p server check

> Task :server:lintReportDebug
Wrote HTML report to file:///home/termii/my/scrcpy/server/build/reports/lint-results-debug.html

BUILD SUCCESSFUL in 10s
45 actionable tasks: 45 executed
[ -d "build-server" ] || ( mkdir "build-server" && \
        meson setup "build-server" --buildtype release -Dcompile_app=false )
The Meson build system
Version: 1.0.1
Source dir: /home/termii/my/scrcpy
Build dir: /home/termii/my/scrcpy/build-server
Build type: native build
Project name: scrcpy
Project version: 2.1.1
C compiler for the host machine: cc (gcc 12.2.0 "cc (Debian 12.2.0-14) 12.2.0")
C linker for the host machine: cc ld.bfd 2.40
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program ./scripts/build-wrapper.sh found: YES (/home/termii/my/scrcpy/server/./scripts/build-wrapper.sh)
Build targets in project: 2

scrcpy 2.1.1

  User defined options
    buildtype  : release
    compile_app: false

Found ninja-1.11.1 at /usr/bin/ninja
ninja -C "build-server"
ninja: Entering directory `build-server'
[0/1] Generating server/scrcpy-server with a custom command
(not invoking gradle, since we are root)
platform-tools-34.0.3 found
SDL2-2.28.0 found
ffmpeg-6.0-scrcpy-4 found
libusb-1.0.26 found
[ -d "build-win32" ] || ( mkdir "build-win32" && \
        meson setup "build-win32" \
                --cross-file cross_win32.txt \
                --buildtype release --strip -Db_lto=true \
                -Dcompile_server=false \
                -Dportable=true )
The Meson build system
Version: 1.0.1
Source dir: /home/termii/my/scrcpy
Build dir: /home/termii/my/scrcpy/build-win32
Build type: cross build
Project name: scrcpy
Project version: 2.1.1
C compiler for the host machine: i686-w64-mingw32-gcc (gcc 12.0.0 "i686-w64-mingw32-gcc (GCC) 12-win32")
C linker for the host machine: i686-w64-mingw32-gcc ld.bfd 2.40
C compiler for the build machine: cc (gcc 12.2.0 "cc (Debian 12.2.0-14) 12.2.0")
C linker for the build machine: cc ld.bfd 2.40
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86
Host machine cpu: i686
Target machine cpu family: x86
Target machine cpu: i686
Windows resource compiler: GNU windres (GNU Binutils) 2.40
Library SDL2 found: YES
Library SDL2main found: YES

app/meson.build:141:4: ERROR: C shared or static library 'avcodec-60' not found

A full log can be found at /home/termii/my/scrcpy/build-win32/meson-logs/meson-log.txt
make: *** [release.mk:72: build-win32] Error 1

If I don't execute the script as root, I get all kinds of errors:

Because you executed with sudo once, so it created the directories with root permissions. You should delete them (as root) and restart (without root).

app/meson.build:141:4: ERROR: C shared or static library 'avcodec-60' not found

When building from Windows, this dependency is downloaded and extracted to app/prebuilt-deps/data. What's the content of this folder (and the ffmpeg one inside it)? If you remove it, it will re-download-extract it.


Allow users to create their own views inside message bubbles.

Pull request checklist

[x] All tests pass. Demo project builds and runs.
[x] I have resolved any merge conflicts.
[x] I have followed the coding style, and reviewed the contributing guidelines. Confirmation: :muscle::sunglasses::facepunch:

This fixes issue #1536.
What's in this pull request?
The JSQCustomMediaItem class is the starting place for allowing users to
create their own custom views. It manages the communication between the
cell/collection view and the custom view.
The JSQCustomMediaView is the foundation for all custom views. Any
custom view either needs to use an instance of this class and add their
own viewes to it or subclass it.
The JSQHTMLMessageView is a subclass of JSQCustomMediaView and allows
HTML to be displayed inside a message bubble. As a nice bonus, the default font and text color for the HTML is fully managed for the user.
Basic unit tests as well as enhancements to the demo are included.

Current coverage is 66.68%

Merging #1537 into develop will increase coverage by +0.03% as of dcc1ecf

@@            develop   #1537   diff @@
=======================================
  Files            63      69     +6
  Stmts          2309    2476   +167
  Branches          0       0
  Methods
=======================================
+ Hit            1539    1651   +112
  Partial           0       0
- Missed          770     825    +55

Review entire Coverage Diff as of dcc1ecf

Powered by Codecov. Updated on successful CI builds.

@ryangrimm this is great. Very nicely done!

At a glance, :+1: :tada:
Thanks for working on this @ryangrimm !
Mind rebasing / resolving conflicts?

@jessesquires thanks! Don't mind rebasing and resolving conflicts at all. Will try to get around to it in the next couple hours or so.

UHUUUUUUU

@jessesquires and @eliburke, I got the conflicts resolved and my branch updated with the latest from develop.

@eliburke it looks like there are conflicts with this again 😥

I'm happy to resolve the conflicts again. @jessesquires have any idea on what it would take to get the pull request accepted so I don't have to resolve conflicts too often?

Ya I'm also having the same issue.

@ryangrimm -- ha 😅 i'm sorry! Actually, maybe hold off for now. After the 7.x release, I'll be merging stuff like crazy.
I'll ping you again when I'm ready to merge if that's ok! 😄

epic Epic EPIC!

When you rotate device multiple times it increasing the size of cell

Better to have params thats specify is that message relates to system or user

What is the status of this pull request? Will this ever get merged?

@jessesquires @ryangrimm I've re-applied all the changes from original @ryangrimm pull request on top of current develop, cause fixing the all the merge conflicts was quite tedious. I've also fixed the issue @ilin-in mentioned:

When you rotate device multiple times it increasing the size of cell

Let me know what you guys think. Would be nice to merge it

Hello everyone!
I'm sorry to inform the community that I'm officially deprecating this project. 😢 Please read my blog post for details:
http://www.jessesquires.com/blog/officially-deprecating-jsqmessagesviewcontroller/
Thus, I'm closing all issues and pull requests and making the necessary updates to formally deprecate the library. I'm sorry if this is unexpected or disappointing. Please know that this was an extremely difficult decision to make. I'd like to thank everyone here for contributing and making this project so great. It was a fun 4 years. 😊
Thanks for understanding,
— jsq


Updated Readme

I saw the readme was out of date, so I decided to update it.
I kept the content pretty much the same, just used org-mode because I prefer it and updated the code to the new format.

There are some problems with your PR and I don't see any real benefit.

it's not written in markdown. org-mode isn't very common and not well supported by many tools, eg. by Flit (the tool that's used for packaging)
in my opinion, the document structure is unintuive. The readme tries to give a small overview of what you need to get started.
in fact, using an object oriented interface is the new format, see #1. The old-style functions are kept for compatibility reasons and simplicity.

I'll add a small phrase about PyPi and a note about the old-style "interface to the old readme. In the future I plan to fix the bad type hints so the source can be used as a good reference to the module's capabilities.
Thank you for your work.


Example in response models

Is there any way to add examples to the response models? I use these examples to show different errors that may occur under the same HTTP status.

This would be useful

gets my vote too


Filter Coinbase account wallets

Description:
Only add sensor entities for accounts with the specified currencies.
This is a none breaking change. If it's not specified then all account wallets will be loaded.
The default behaviour gives the same results as it does now.
A simple configuration would make all data available for the UI.
Currently there's a discussion (https://github.com/home-assistant/architecture/issues/100)
about removing monitored conditions altogether.
If the user specifies the key without a value (empty list),
then no sensor entities are added for the wallet (e.g. a user only wants exchange rates).
Related issue (if applicable): fixes...
Currently you can just customize the entities to hide them.
However, Coinbase regularly adds new account wallets with new currencies.
You would always need to customize these new entities
and there's no way to get them out of the entity registry itself.
However, this is something I would prefer since I don't need all my 'bank accounts' in HA either
(and there's no point in adding them when the wallet is empty anyway).
Pull request in home-assistant.io with documentation (if applicable): https://github.com/home-assistant/home-assistant.io/pull/7412
Example entry for configuration.yaml (if applicable):
coinbase:
  api_key: !secret coinbase_api_key
  api_secret: !secret coinbase_api_secret
  account_balance_currencies:
    - EUR
    - BTC

Checklist:

[x] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] New dependencies have been added to the REQUIREMENTS variable (example).
[ ] New dependencies are only imported inside functions that use them (example).
[ ] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[ ] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

I guess it's fine as 100 is not decided yet. See no docs PR

Thanks for the reminder, it's now added.


[Enhancement] Allow editing of multiple markers

Adding Build/Clear buttons as well.
Don't know how the property drawer for the datablock select will function

Build button will be renamed to Build Random
Also this button will be added when selecting one marker

Fixed. Don't mind the button thats supposed to let you choose a datablock not existing. Still have to apply that change in my build.


Add new transpiler passes to generate single-pulse RX calibrations on the fly

Summary
Two new transpiler passes are added to generate single-pulse RX gate calibrations on the fly. These single-pulse RX calibrations will reduce the gate time in half, as described in P.Gokhale et al, arXiv:2004.11205.

qiskit.transpiler.passes.optimization.normalize_rx_angle.NormalizeRXAngle

Inherits TransformationPass
Performs three optimizations to reduce the amount of RX calibration data that needs to be generated:

Wrapping RX gate rotation angles to [0, pi]: Replaces an RX gate with a negative rotation angle, RX(-theta), with a sequence: RZ(pi)-RX(theta)-RZ(-pi). This will help reduce the size of calibration data, as we won't have to keep separate, phase-flipped calibrations for negative rotation angles.
Replacing RX(pi/2) and RX(pi) with SX and X gates: This will allow us to exploit the more accurate, hardware-calibrated pulses.
Quantizing the rotation angles using a user-provided resolution: If the resolution is set to 0, this pass will not perform any quantization.

Required to be ran before the qiskit.transpiler.passes.calibration.rx_builder.RXCalibrationBuilder

qiskit.transpiler.passes.calibration.rx_builder.RXCalibrationBuilder

Inherits CalibrationBuilder
Generates RX calibrations on the fly. The pulse calibrations are bootstrapped from the SX gate calibration in the target. The amplitude is linearly scaled to achieve the desired arbitrary rotation angle.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: Jaeun Kim:x: jaeunkimJaeun Kim seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

I'm trying to figure out how to remove "Jaeun Kim" from the contributor list as signing CLA with the deactivated IBM account is impossible.

You'll have to edit the git history on your pull request's branch. You can combine git rebase -i to interactively edit the history and git commit --amend --reset-author to change the author name and email for the commits.