[Bug] (v0.3.6.post2) Output degredation when using structured output

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[X] 2. The bug has not been fixed in the latest version.
[X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
[X] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
[X] 5. Please use English, otherwise it will be closed.

Describe the bug
The results (w and w/o JSON schema) are different, while those generated from vllm server (v0.6.4.post1) remain the same
Reproduction
How to start sglang server
services:
  llm-sglang-dev:
    image: lmsysorg/sglang:latest
    container_name: llm-sglang-dev
    restart: unless-stopped
    environment:
      HUGGING_FACE_HUB_TOKEN: <my-hf-token>
    ports:
      - "8007:8007"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    env_file:
      - .env
    command: >
      python3 -m sglang.launch_server
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --host <IP_ADDRESS>
      --port 8007
      --api-key <my-api-key>
      --served-model-name gpt-4o
      --tensor-parallel-size 1
      --mem-fraction-static 0.4
      --random-seed 42
      --enable-p2p-check
      --show-time-cost
      --quantization awq_marlin
      --grammar-backend xgrammar
      --enable-cache-report
      --context-length 2048

How to start vllm server
services:
  llm-vllm:
    image: vllm/vllm-openai:latest
    container_name: llm-vllm
    restart: unless-stopped
    environment:
      HUGGING_FACE_HUB_TOKEN: <my-hf-token>
    ports:
      - "8007:8007"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --host <IP_ADDRESS>
      --port 8007
      --api-key <my-api-key>
      --max-model-len 16382
      --tensor-parallel-size 1
      --gpu-memory-utilization 0.8
      --served-model-name gpt-4o
      --seed 42
      --disable-log-requests
      --enable-prefix-caching
      --model MODEL=Qwen/Qwen2.5-7B-Instruct-AWQ

Python script
import json
import openai

from pydantic import BaseModel

client = openai.OpenAI(
    base_url="http://localhost:8007/v1",
    api_key="Lizai@54321"
)

class Players(BaseModel):
    names: list[str]

class Model(BaseModel):
    name: str
    number_of_parameters: str
    number_of_max_tokens: str
    architecture: list[str]

class Usage(BaseModel):
    use_case: list[str]
    license: str

class Schema(BaseModel):
    model: Model
    usage: Usage

document = """We introduce Mistral 7B, a 7–billion-parameter language model engineered for
superior performance and efficiency. Mistral 7B outperforms the best open 13B
model (Llama 2) across all evaluated benchmarks, and the best released 34B
model (Llama 1) in reasoning, mathematics, and code generation. Our model
leverages grouped-query attention (GQA) for faster inference, coupled with sliding
window attention (SWA) to effectively handle sequences of arbitrary length with a
reduced inference cost. We also provide a model fine-tuned to follow instructions,
Mistral 7B – Instruct, that surpasses Llama 2 13B – chat model both on human and
automated benchmarks. Our models are released under the Apache 2.0 license.
Code: <https://github.com/mistralai/mistral-src>
Webpage: <https://mistral.ai/news/announcing-mistral-7b/>"""

template = """{
    "model": {
        "name": "",
        "number_of_parameters": "",
        "number_of_max_tokens": "",
        "architecture": []
    },
    "usage": {
        "use_case": [],
        "licence": ""
    }
}"""

schema = json.dumps(json.loads(template))

completion = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": f"Task: Extract precise values for each field in the provided output schema from the given document.\nInstructions:\n-Do not hallucinate, paraphrase, or modify the extracted values.\n- If a field has no corresponding value in the document, leave it as an empty string (\"\").\n\nDocument:\n{document}\nSchema:\n{schema}"}
    ],
    temperature=0.0,
    max_tokens=256,
    extra_body={
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": Schema.__name__,
                "schema": Schema.model_json_schema()
            }
        }
    }
)
print(completion.choices[0].message.content)

Results without json_schema
vllm
{
  "model": {
    "name": "Mistral 7B",
    "number_of_parameters": "7 billion",
    "number_of_max_tokens": "",
    "architecture": ["grouped-query attention (GQA)", "sliding window attention (SWA)"]
  },
  "usage": {
    "use_case": ["superior performance and efficiency", "reasoning", "mathematics", "code generation", "following instructions"],
    "licence": "Apache 2.0"
  }
}

sglang
{
  "model": {
    "name": "Mistral 7B",
    "number_of_parameters": "7 billion",
    "number_of_max_tokens": "",
    "architecture": ["grouped-query attention (GQA)", "sliding window attention (SWA)"]
  },
  "usage": {
    "use_case": ["superior performance and efficiency", "reasoning", "mathematics", "code generation", "following instructions"],
    "licence": "Apache 2.0"
  }
}

Results with json_schema
vllm
{
  "model": {
    "name": "Mistral 7B",
    "number_of_parameters": "7 billion",
    "number_of_max_tokens": "",
    "architecture": ["grouped-query attention (GQA) for faster inference", "sliding window attention (SWA)"]
  },
  "usage": {
    "use_case": ["superior performance and efficiency", "reasoning", "mathematics", "code generation", "following instructions"],
    "license": "Apache 2.0"
  }
}

sglang (xgrammar backend)
{"model": {"name": "Mistral 7B", "number_of_parameters": "7–billion", "number_of_max_tokens": "", "architecture": ["grouped-query attention (GQA)", "sliding window attention (SWA)"]}, "usage": {"use_case": [], "license": "Apache 2.0"}}

sglang (outlines backend)
{"model": {"name": "Mistral 7B", "number_of_parameters": "7–billion", "number_of_max_tokens": "", "architecture": ["grouped-query attention (GQA)", "sliding window attention (SWA)"]}, "usage": {"use_case": [], "license": "Apache 2.0"}}

Environment
Python: 3.10.15 (main, Sep  7 2024, 18:35:33) [GCC 9.4.0]
CUDA available: True
GPU 0: NVIDIA A10G
GPU 0 Compute Capability: 8.6
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.105
CUDA Driver Version: 550.120
PyTorch: 2.5.1+cu124
flashinfer: 0.1.6+cu121torch2.4
triton: 3.1.0
transformers: 4.46.3
torchao: 0.6.1
numpy: 1.26.4
aiohttp: 3.11.7
fastapi: 0.115.5
hf_transfer: 0.1.8
huggingface_hub: 0.26.2
interegular: 0.3.3
psutil: 6.1.0
pydantic: 2.10.1
multipart: 0.0.17
zmq: 26.2.0
uvicorn: 0.32.1
uvloop: 0.21.0
vllm: 0.6.4.post1
openai: 1.55.1
anthropic: 0.39.0
NVIDIA Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-47    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hypervisor vendor: KVM
ulimit soft: 1048576

cc @merrymercy

Thanks for the reporting this. I can reproduce this error. The problem is that this model works better with multi-line style JSON but the default argument in sglang uses single-line style. We can fix this for both outlines and xgrammar backend.
Fix for outlines backend (works)
You can add --constrained-json-whitespace-pattern "[\n\t ]*" when you launch the server. See also #1438
python3 -m sglang.launch_server --model Qwen/Qwen2.5-7B-Instruct-AWQ --port 8007 --quantization awq_marlin  --grammar-backend outlines --constrained-json-whitespace-pattern "[\n\t ]*"

Output
{
  "model": {
    "name": "Mistral 7B",
    "number_of_parameters": "7 billion",
    "number_of_max_tokens": "",
    "architecture": ["grouped-query attention (GQA)", "sliding window attention (SWA)"]
  },
  "usage": {
    "use_case": ["superior performance and efficiency", "reasoning", "mathematics", "code generation", "following instructions"],
    "license": "Apache 2.0"
  }
}

Fix for xgrammar backend (does not work)
Try this commit https://github.com/sgl-project/sglang/commit/dd4482e3c796437c4ea67a80ed685667ad8ff947
python3 -m sglang.launch_server --model Qwen/Qwen2.5-7B-Instruct-AWQ --port 8007 --quantization awq_marlin  --grammar-backend xgrammar

Output
{
  "model": {
    "name": "Mistral 7B",
    "number_of_parameters": "7 billion",
    "number_of_max_tokens": "",
    "architecture": []
  },
  "usage": {
    "use_case": [],
    "license": "Apache 2.0"
  }
}

However, this fix does not work. I think there are some subtle details on how it handle the whitespace @Ubospica .

I am also facing a wierd issue when using xgrammar as backend. Not sure if this is related.
I am using document prefix caching to do multiple extractions at the same time. Some of them use structured json output. And some of them outputs plain text. When using xgrammar using sgl.gen with json_schema, the outputs in plain text are changing, and sometimes not even terminating.  Wierd thing is the plain text is not using json_schema.
While using outlines, its working as expected.
Isnt sgl.gen with json_schema and xgrammar not supported yet?
I can provide more information to reproduce if necessary (but its custom code).
Thanks,

I also observe in my json output tests with around 1500 requests that the xgrammar backend allows for infinite generation. It also does not adhere to the api max_tokens limit and continues generating until memory overflow occurs, after which the generation is abruptly cut off. This does not happen with outlines.

@arunpatala @Swipe4057 any minimal small reproducible examples will be very helpful here. The developers of grammar @Ubospica are here ready to help if the bugs can be easily reproduced.

@merrymercy Hi, thanks for your advice. I pulled the latest version v0.4.0 and added --constrained-json-whitespace-pattern "[\n\t ]*". Both xgrammar and outlines backend output the same result.
'{"model": {"name": "Mistral 7B", "number_of_parameters": "7–billion", "number_of_max_tokens": "", "architecture": ["grouped-query attention (GQA)", "sliding window attention (SWA)"]}, "usage": {"use_case": [], "license": "Apache 2.0"}}'

@Quang-elec44 For XGrammar, we found the reason is XGrammar requires LLM generate in strictly formatted JSON. This is not strictly formatted JSON (the array is compressed in one line):
"usage": {
  "use_case": ["superior performance and efficiency", "reasoning", "mathematics", "code generation", "following instructions"]
}

This is strictly formatted:
"usage": {
  "use_case": [
    "superior performance and efficiency",
    "reasoning",
    "mathematics",
    "code generation",
    "following instructions"
  ]
}

However, this strict requirement sometimes makes LLM's output quality deteriorate. In this case, the LLM will generate
"usage": {
  "use_case": []
}

which is still strictly formatted, but not meaningful.
We will relax this restriction and allow non-strictly formatted json in the real case in a recent version to ensure the output quality.

@merrymercy It seems I was able to understand a bit more in detail; the issue isn't with xgrammar, but rather that my LLM doesn't generate stop tokens under certain parameters.

Is there a way to format the json data for finetuning to follow the format of xgrammar?

After upgrading SGlang, JSON schema completely stopped working for me (breaking changes, not mentioned anywhere!)
response_format.json_schema
  Input should be a valid dictionary or instance of JsonSchemaResponseFormat [type=model_type, input_value='{\n  "$schema": "http://...\n  "type": "object"\n}', input_type=str]

Looks like it is not possible to pass schema as a string anymore. Ok fine, I pass it as object, but still get an error
response_format.json_schema.name
  Field required [type=missing, input_value={'$schema': 'http://json-...ema#', 'type': 'object'}, input_type=dict]

Why does a schema require a name???
Even after adding the name, the schema was fully ignored and the generated output was not following the schema at all without any errors in the logs:

The problem of XGrammar mentioned above (https://github.com/sgl-project/sglang/issues/2216#issuecomment-2516192009) is solved in XGrammar v0.1.6. It is also updated in SGLang (#2390).
This should solve the issue mentioned by @Quang-elec44. @arunpatala could you try it again after updating to the latest version of XGrammar? I think the output quality should be better, and that may have fixed your problem.

I tried to reproduce the error when i am mixing structured generation with normal generation using sgl forks.
The example task is to extract details from a abstract of a paper.

abstract = """
Computer Science > Computer Vision and Pattern Recognition
[Submitted on 5 Dec 2024]
PaintScene4D: Consistent 4D Scene Generation from Text Prompts
Vinayak Gupta, Yunze Man, Yu-Xiong Wang
Recent advances in diffusion models have revolutionized 2D and 3D content creation, yet generating photorealistic dynamic 4D scenes remains a significant challenge. Existing dynamic 4D generation methods typically rely on distilling knowledge from pre-trained 3D generative models, often fine-tuned on synthetic object datasets. Consequently, the resulting scenes tend to be object-centric and lack photorealism. While text-to-video models can generate more realistic scenes with motion, they often struggle with spatial understanding and provide limited control over camera viewpoints during rendering. To address these limitations, we present PaintScene4D, a novel text-to-4D scene generation framework that departs from conventional multi-view generative models in favor of a streamlined architecture that harnesses video generative models trained on diverse real-world datasets. Our method first generates a reference video using a video generation model, and then employs a strategic camera array selection for rendering. We apply a progressive warping and inpainting technique to ensure both spatial and temporal consistency across multiple viewpoints. Finally, we optimize multi-view images using a dynamic renderer, enabling flexible camera control based on user preferences. Adopting a training-free architecture, our PaintScene4D efficiently produces realistic 4D scenes that can be viewed from arbitrary trajectories. The code will be made publicly available. Our project page is at this https URL
Comments:	Project page: this https URL
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2412.04471 [cs.CV]
 	(or arXiv:2412.04471v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2412.04471
Focus to learn more
Submission history
From: Yunze Man [view email]
[v1] Thu, 5 Dec 2024 18:59:57 UTC (7,775 KB)

"""

from pydantic import BaseModel
from typing import List, Dict

class SubmissionHistory(BaseModel):
    version: str
    date: str
    file_size: str

class Metadata(BaseModel):
    comments: str
    submission_history: SubmissionHistory
    doi: str
    arxiv_id: str

class Abstract(BaseModel):
    title: str
    authors: List[str]
    submission_date: str
    categories: List[str]
    summary: str
    metadata: Metadata

import sglang as sgl

abstract_prompt = """
**ABSTRACT**
{}

"""

abstract_instruction = """
**INSTRUCTION:**

"Parse the provided abstract and metadata of a research paper into JSON format with the following structure:

1. Include the `title`, `authors`, `submission_date`, and `categories` as direct fields.
2. Exclude the `abstract` field.
3. Add a `summary` field that concisely explains the core idea, methodology, and significance of the paper.
4. Retain a `metadata` field containing any additional details such as `comments`, `submission_history`, `doi`, and `arxiv_id`.

Ensure the JSON is well-structured and adheres to the specified format."

"""

@sgl.function
def gets(s, abstract, keys):
    s += sgl.user_begin()
    s += abstract_prompt.format(abstract)
    forks = s.fork(1 + len(keys))
    forks[0] += abstract_instruction + sgl.user_end()
    forks[0] += sgl.assistant(sgl.gen("response", json_schema=Abstract.schema_json(), max_tokens=1024, temperature=0.0))
    for k,f in zip(keys, forks[1:]):
        f += f"Extract the following field: {k}" + sgl.user_end()
        f += sgl.assistant(sgl.gen("response", max_tokens=1024,
                                   temperature=0.0))
    forks.join()
    s["return1"] = forks[0]["response"]
    s["return2"] = [f["response"] for f in forks[1:]]

sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))

keys = ['title', 'authors', 'summary']

states = gets.run(abstract, keys)

for i,k in zip(keys, states["return2"]):
    print(i, ":")
    print(k)

I am running sglang server with
python3 -m sglang.launch_server \
        --context-length 8192 \
        --served-model-name model \
        --host <IP_ADDRESS> --port 30000 \
        --mem-fraction-static 0.85 \
        --max-running-requests 64 \
        --model-path meta-llama/Llama-3.2-3B-Instruct \
        --grammar-backend xgrammar

I tried with both the xgrammar v0.1.5 and v0.1.6 with latest main branch in docker.
I am getting the following output for outlines at temperature 0.0 and 0.25
# outlines temperature=0.0

--------
title :
PaintScene4D: Consistent 4D Scene Generation from Text Prompts
--------
authors :
The authors of the paper are:

1. Vinayak Gupta
2. Yunze Man
3. Yu-Xiong Wang
--------
summary :
Here is the extracted summary:

PaintScene4D: Consistent 4D Scene Generation from Text Prompts

#outlines temperature=0.25

--------
title :
PaintScene4D: Consistent 4D Scene Generation from Text Prompts
--------
authors :
The authors of the paper are:

1. Vinayak Gupta
2. Yunze Man
3. Yu-Xiong Wang
--------
summary :
Here is the extracted summary:

**PaintScene4D: Consistent 4D Scene Generation from Text Prompts**

Recent advances in diffusion models have revolutionized 2D and 3D content creation, yet generating photorealistic dynamic 4D scenes remains a significant challenge.

But the output of xgrammar at especially lower temperatures are not generated properly.

#xgrammar temperature=0

--------
title :
PaintScene4D: Consistent 4!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--------
authors :
The authors of the paper are:

1.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--------
summary :
Here is the extracted summary:

PaintScene4!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

xgrammar temperature=0.25

--------
title :
assistant

PaintScene4D: Consistent 4D Scene Generation from Text Prompts
--------
authors :
assistant

The authors of the paper are:

1. Vinayak Gupta
2. Yunze Man
3. Yu-Xiong Wang
--------
summary :

Also xgrammar is generating "assistant" keyword at temperature 0.25 sometimes or not generating at all. The structured output seems to be all right for both grammar backends.
Hope this helps to reproduce the bug. Let me know if any other information is needed.

@arunpatala Thanks for producing the reproducible script! I think that should be a problem in the mask application process where the mask is applied on non-structured requests. We will fix that problem soon.

@Ubospica thanks for looking into it.


Might be down

Hi everyone,
I know this project is not currently maintained, but I deployed one of my first decent projects while still learning web dev using Thin, so I'm interested in understanding whether Thin has been shut down or if there is a bug or internal error. This information will help me decide whether to retire my project, which is part of my portfolio.
When I go to the "Data" section in the dashboard, I encounter the following error:

Additionally, when trying to log in to the project, I see this error:

I loved Thin; it was a great idea and project. I wish you all the best!

Yes we shut down the server for now as it's too expensive to run (and we ran out of free aws credits)


CSS-3963: Add tutorial and docs

A rigorously conceived and written end-to-end tutorial to help new users. I used the Data Platform template used in their charms (e.g. PostgreSQL K8s) for inspiration.
Note for reviewers: a lot of the tutorial pages are just parts of the README split up (and outputs of juju status) into different pages for clarity.
Once approved, I will create the necessary Discourse pages to reflect these changes on the charm's Charmhub page.

Please re-request review when you are done with the changes, I had my review canceled because of a change while I was doing it.

Will do! The PR is meant to be for the tutorial mainly, but some library changes have caused the integration tests to break, so I have been trying to fix them.


Bug: NPEx when using serverId

on my POM I have this
	<build>
...
		<plugins>
			<plugin>
				<groupId>com.spotify</groupId>
				<artifactId>docker-maven-plugin</artifactId>
				<version>0.2.8</version>
					<configuration>
						<serverId>docker-hub</serverId>
...
					</configuration>
			</plugin>
		</plugins>
	</build>

under my settings.xml  (xpath //settings/servers/server) I have this:
    <server>
    	<id>docker-hub</id>
    	<username>******</username>
    	<password>*******</password>
    </server>

when running mvn clean install docker:build I get this errors
[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:0.2.8:build (default-cli) on project k2: Exception caught: NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.spotify:docker-maven-plugin:0.2.8:build (default-cli) on project k2: Exception caught
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Exception caught
	at com.spotify.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:117)
	at com.spotify.docker.BuildMojo.execute(BuildMojo.java:79)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
	... 19 more
Caused by: java.lang.NullPointerException
	at com.spotify.docker.AbstractDockerMojo.getEmail(AbstractDockerMojo.java:155)
	at com.spotify.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:86)
	... 22 more

I just ran into this NPE.
Its blocking our ability to use this Maven plugin with our Jenkins CI builds.
We use Config File Provider Plugin and Credentials Plugin for managing the Maven settings.xml file in our build farm.  Those plugins will stitch credentials into a settings.xml at the beginning of a build, but the XML stanzas they stitch into the settings.xml file look like this:
<server>
<id>server_id</id>
<username>...</username>
<password>...</password>
</server>
No <configuration><email>...</email></configuration> entries are created.
Perhaps there should be a way to loosen the check on whether an email needs to be present in the settings.xml file for a docker push operation?

@aharonha @greglanthier Thanks for the bug report.
I believe docker needs the email. The email, password, and username are hashed together and the has is also sent to the server during authentication. Would it help to make the error message more obvious?

A more obvious error would be useful, sure.
In our CI environment when we do a "docker push" we only login with a usename & password.  Perhaps that isn't best practice...
Regardless, the Jenkins configuration plugins I mentioned don't let you specify additional configuration XML stanzas in the <server .../> elements, and since Jenkins drives our entire CI / CD process I'm trying to find a happy compromise between Maven & Jenkins in this case.
Thanks for looking at this so quickly!


Maintainer vs. backer profile shown on pledge page

From the new backer dashboard if I enter an issue link, I'm redirected to the pledge page, but the state of the profile selector defaults to the maintainer vs. current active. We should set this in our Zustand store to carry it over.

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog :^)

I think that the bug here is in the profile picker, and the fact that you're pledging to an issue in an org that you're a maintainer of.

You can verify this by viewing the pledge page of an org that you're not a maintainer of, like: https://polar.sh/TanStack/query/issues/5913
I think that the public pages should always be "used" as your backer profile, and never as an organization.
This could be handled by the useCurrentOrgAndRepoFromURL hook, by making it behave differently on "public" vs "maintainer" pages. As a bonus, we don't need to add any state to handle this. ;-)

Great clarification and callout @zegl! Completely agree that the public pages should always be used as the backer profile and like the recommended approach of setting that rule to avoid having additional state.


Why do you use the different vad method for preparing data for  d-vector and for uis-rnn?

In your code, librosa.effects.split(utter, top_db=30) for d-vector and webrtcvad for uis-rnn.

I use librosa for preprocessing because a VAD isn't needed. You can also try a VAD.


Relationship exists in browser, not in fetched object

In my Swift project I am using Realm 0.96.2 (using Cocoapods).
My models:
class Game: Object {
    dynamic var id = 0
    dynamic var name = ""
    var activities = List<Activity>()

    override static func primaryKey() -> String {
        return "id"
    }
}

class Activity: Object {
    dynamic var id = 0
    dynamic var name = ""

    override static func primaryKey() -> String {
        return "id"
    }
}

class Request: Object {

    dynamic var id: Int = 0
    dynamic var message = ""
    dynamic var activity: Activity?
    dynamic var game: Game?

    override static func primaryKey() -> String {
        return "id"
    }
}

The Game and Activity object are imported from a JSON response using ObjectMapper. After the import is done my Game object has a couple of Activity objects connected to them (this is also visible in the Realm Browser.
Now when I create a Request object and I assign an Activity and a Game to it they both seem to stay empty. The weird thing is, when I po request (which is a Request object) I actually see the properties filled and all, but at the same time in the Variables View of Xcode those two properties are empty.
Perform po request:
(lldb) po self.requests[0]
Request {
	activity = Activity {
		id = 1;
		name = Full run;
		group = Kings Fall Heroic;
		permalink = king-heroic-full;
	};
	game = Game {
		id = 1;
		name = Destiny;
		activities = RLMArray <0x7ffdfbdb22e0> (
		);
	};
	id = 27;
	message = ;
}

And here is a screenshot of the variable view of Xcode, I know it shows more properties, I kept filtering them out in my text here, but filtering from screenshots is pretty hard ;)

Also, is it weird that the game property from my Request doesn't contain any Activity objects, while the ARE there in the database.
I made my associated properties mutable instead of immutable (var activities = List<Activity>()) because of the ObjectMapper requirements.

Ok, shut this down, always after posting I reanalyze my code and see that my ObjectMapper overwrites properties of nested objects while importing the Request object.


Environment Scope

Could be nice to add Environment Scope - default and loaded from the environment variables.
final env = const String.fromEnvironment('env', defaultValue: '').toLowerCase();

Closed by #57. Thanks for the issues!

Reopened to add the environment env variable getter

@purplenoodlesoop
Could be nice to app enum or predefined environments:
Development
Environment
Production
Staging

Closed again by #58


changing resolution causes a single very short pulse

Hi @Dlloydev,
I was experimenting with using ledc to control a servo, and I found that using ledcSetup seems to cause a pulse to be output on an attached pin. It's more of an issue for servo control (it causes a twitch) than the use case of analogWrite (it wouldn't affect controlling an led), but I thought I'd let you know about this in case you plan to output servo signals (actually, it would be cool if the library would have a function that makes it easy to output servo signal PWM, but that's off topic now).
Here's the code I used:
#include "analogWrite.h"

#define analogPin 33 //pin to analogWrite to
#define triggerPin 26 //pin that triggers oscilloscope when resolution changed

void setup() {
  pinMode(triggerPin,OUTPUT);
  digitalWrite(triggerPin,LOW);
  analogWrite(analogPin, 50);
  delay(1000); //wait a bit to get some pulses at the first resolution
  digitalWrite(triggerPin,HIGH); //trigger oscilloscope
  analogWriteResolution(analogPin, 10); //change resolution
}
void loop(){
  delay(1);
}

it produced the following signal:

I found a potential fix for this issue: I was able to change frequency with ledcSetup without getting the pulse by using ledcDetachPin, then calling ledcSetup, and then calling ledcAttachPin.
Since the pulse only happens once when changing resolution this is really not at all a critical bug to fix, but I thought you might want to know about it.
Thank you for spending so much time making great opensource libraries!
-joshua-8

Thank you for the detailed info and and fix!
I've pushed a commit with your fix (tested OK on my ESP32 Dev Module).


[Event] Add method to request calendarView from Office365 API

Since the getlist do not return all recurring instances of an event,
calendarView gives a way to get the occurrences, exceptions, and single
instances of events

looks good. Please remove the comments about the date and it should be mergeable


Update to work with new log_list.json endpoint

Log list is now at https://www.gstatic.com/ct/log_list/v3/log_list.json and old url is no longer valid. JSON structure is different and urls contain "https://", so processing is updated to account for this.
Also, version bump.

Also, version bump.

I think that a commit changing something should not bump the version


Eliminate needless lifetimes

Result from __CARGO_FIX_YOLO=1 cargo clippy --fix.
As newest nightly clippy bleats about all of this.

I've been avoiding doing this to any repo as I think it is worse and loses useful information.
I'd much rather just suppress it. There's an open issue in clippy about separating this or changing it as well.

https://github.com/rust-lang/rust-clippy/issues/13514 is that issue.

I see your point, though I think it's fine for all the cases in this PR though? But I have no strong opinion here...

I don't really want to suppress this lint, because it does give useful results for e.g. functions.
That being said, these changes are also worse... https://github.com/rust-lang/rust-clippy/pull/13286 seems to just be an unfortunate breaking change.

That being said, these changes are also worse...

You mean those in this PR?
So I think it comes down to sooner or later disable that lint or subdue to it (or add alternatives to clippy as suggested in this issue).
Should I merge this or close it?

Will leave it up to Daniel but in the other repos, my inclination is to suppress this rather than do these sorts of changes.

If you're going to be running nightly, you should probably supress it locally until the story on the clippy side becomes clearer.

Sure that's not the issue, I can suppress it locally. So rather close this PR for now? As at least two people seem to be opposed, and I don't really care, but I like less syntax if possible (without too much loss of information).

I think so; we should see what decision is made on the clippy side first.

So I guess in anticipation to updating the Rust version I should merge this?

Yeah, that would be good.


Abort in dump() method

What is the issue you have?

Eventually jsonObject.dump() crashes as follows:
#0  0x00007f30680cd428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f30680cf02a in __GI_abort () at abort.c:89
#2  0x00007f30681152e8 in __malloc_assert (
    assertion=assertion@entry=0x7f3068229190 "(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)",
    file=file@entry=0x7f3068225bc5 "malloc.c", line=line@entry=2394, function=function@entry=0x7f30682299d8 <__func__.11510> "sysmalloc") at malloc.c:301
#3  0x00007f3068119426 in sysmalloc (nb=nb@entry=1936, av=av@entry=0x7f306845cb20 <main_arena>) at malloc.c:2391
#4  0x00007f306811a743 in _int_malloc (av=av@entry=0x7f306845cb20 <main_arena>, bytes=bytes@entry=1921) at malloc.c:3827
#5  0x00007f306811c184 in __GI___libc_malloc (bytes=1921) at malloc.c:2913
#6  0x00007f3068922e78 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f30689b4499 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f30689b5833 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00000000005c9e69 in nlohmann::detail::output_string_adapter<char, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::write_characters (this=0x1ebd020,
    s=0x7ffd7146c16a "dtlsState65:FE:25:45:29:D2:87:D8:22:03:0A:54:5E:61:19:27:30:B8:5D:C0:97:B0:D3:E1:D9:CF:CD:EB:E5:19:44:F1:57:84:32:12:21:B6:06:1E:39:03:59:85:63:26:A8:10:B7:51:1E:5C:EF:E7:BB:51:CE:70:1E:6B:60", length=9)
    at ../deps/json/single_include/nlohmann/json.hpp:11274
Python Exception <class 'gdb.error'> There is no member named _M_dataplus.:
#10 0x00000000005cb051 in nlohmann::detail::serializer<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> >::dump_escaped (this=0x7ffd7146c110, s=, ensure_ascii=false) at ../deps/json/single_include/nlohmann/json.hpp:14251
#11 0x00000000005c88f2 in nlohmann::detail::serializer<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> >::dump (this=0x7ffd7146c110, val=..., pretty_print=false, ensure_ascii=false, indent_step=0, current_indent=0) at ../deps/json/single_include/nlohmann/json.hpp:13897
#12 0x00000000005c8956 in nlohmann::detail::serializer<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> >::dump (this=0x7ffd7146c110, val=..., pretty_print=false, ensure_ascii=false, indent_step=0, current_indent=0) at ../deps/json/single_include/nlohmann/json.hpp:13899
#13 0x00000000005c7385 in nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>::dump (
    this=0x7ffd7146c540, indent=-1, indent_char=32 ' ', ensure_ascii=false, error_handler=nlohmann::detail::error_handler_t::strict) at ../deps/json/single_include/nlohmann/json.hpp:16574
#14 0x00000000005d0938 in Channel::UnixStreamSocket::Send (this=0x1e647f0, jsonMessage=...) at ../src/Channel/UnixStreamSocket.cpp:48

Please describe the steps to reproduce the issue. Can you provide a small but working code example?

Working on it, but it's hard since it happens randomly.

Which compiler and operating system are you using? Is it a supported compiler?

It does not happen in OSX with Clang. It happens in Linux Debian with gcc 6.3.0 and Ubuntu with 5.4.0 in Docker.

Did you use a released version of the library or the version from the develop branch?

Latest stable 3.7.3

This seems to be terrible IMHO:
#9  0x00000000005c9e69 in nlohmann::detail::output_string_adapter<char, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::write_characters (this=0x1ebd020,
    s=0x7ffd7146c16a "dtlsState65:FE:25:45:29:D2:87:D8:22:03:0A:54:5E:61:19:27:30:B8:5D:C0:97:B0:D3:E1:D9:CF:CD:EB:E5:19:44:F1:57:84:32:12:21:B6:06:1E:39:03:59:85:63:26:A8:10:B7:51:1E:5C:EF:E7:BB:51:CE:70:1E:6B:60", length=9)
    at ../deps/json/single_include/nlohmann/json.hpp:11274

dtlsState is a root object key and it's value is "connected", "failed" etc. There is a memory corruption here somehow. In fact, eventually I see this when running the binary in a unit test:
*** Error in `/mediasoup/worker/out/Release/mediasoup-worker': corrupted size vs. prev_size: 0x0000000001ff6d60 ***

Let's put this issue on hold. It may be a change introduced in our code that may generate some memory corruption somewhere else.

Bug in our side. Sorry for the noise.


Support PyQt 5.12.3

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Resolves #58.
I don't have a Windows machine, so I can't test the Win build. Tested locally on a Linux machine (I have neither root privilege on it nor control over its Conda root environment) against the recent Qt 5.12 package (conda-forge/qt-feedstock#110).
Note that a private copy of the sip module is necessary for PyQt 5.11+ (conda-forge/sip-feedstock#16), unless changes are made on the sip-feedstock side.
ps. I didn't realize until now that there're parallel efforts (#59, #60) 😦 But, this PR demonstrates how to resolve the PyQt5.sip not found issue (in a brute force way...) Hope this helps.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-webservice.
I tried to re-render for you, but it looks like there was nothing to do.

ps. Please feel free to take over or close this PR as you see fit. I'm happy as long as a newer package can be built.

Thanks for the tip about PyQt5.sip. I'll take into account to finish #59.

Hi! This is the friendly automated conda-forge-linting service.
I was trying to look for recipes to lint for you, but it appears we have a merge conflict.
Please try to merge or rebase with the base branch to resolve this conflict.
Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-webservice.
I tried to re-render for you, but it looks like there was nothing to do.

Unless they changed something upstream, the PyQt configure.py will not be able to properly link QtDesigner with the Python library without 0003-configure.py...  You will probably get the plugin but it will be invalid without the patch.

@isuruf @ccordoba12 Linux & Mac OS X are built successfully, but on Windows I kept seeing LINK error (missing or cannot open shell32.exe):
LINK : fatal error LNK1181: cannot open input file 'C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64\shell32.lib'

NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\link.EXE"' : return code '0x49d'

Stop.

NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\nmake.exe"' : return code '0x2'

Stop.

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=73750&view=logs&jobId=171a126d-c574-5c8c-1269-ff3b989e923d&taskId=7225e372-3a5e-5b11-c153-719d50bbeacd&lineStart=583&lineEnd=591&colStart=1&colEnd=6
Do you have experience on solving this?

Unless they changed something upstream, the PyQt configure.py will not be able to properly link QtDesigner with the Python library without 0003-configure.py... You will probably get the plugin but it will be invalid without the patch.

@hhslepicka I thought the 0003 patch was for 5.9.x? When I inspected configure.py I didn't find the relevant code to which the patch refers. Advice?

Alright, I have no idea how to fix the Windows error due to lack of understanding. Remove the WIP flag (since it works on unix-like platforms) and let admins or anyone interested take over this PR.

Unless they changed something upstream, the PyQt configure.py will not be able to properly link QtDesigner with the Python library without 0003-configure.py... You will probably get the plugin but it will be invalid without the patch.

@hhslepicka I thought the 0003 patch was for 5.9.x? When I inspected configure.py in 5.12.3 I didn't find the relevant code to which the patch refers. Advice?

@leofang check the configure.py starting at line 665...
Probably you will need to update the patch or at least try to apply it and check for failure.

@hhslepicka OK that's a good catch. I'll restore the patches when I have time. But let me note that the builds for Linux and OS X with py3x have been successful in ebec4b5, long before I removed the patches, while Windows has kept failing regardless of the patches are applied or not.

@leofang, that is true... the build will succeed even without the patches. They fix runtime issues.
The 0003-configure patch fixes the python library that is linked and the other patch it is my understanding that it fixes some extra path that was being added with Windows.
The builds will succeed but without the patches some extensions of PyQt will not be usable.
Thank you for working on that. I will try to take a look at the windows if I have some time later on the week.

Dear @hhslepicka,

The builds will succeed but without the patches some extensions of PyQt will not be usable.

I am curious, would these potential failures you referred to be caught by import tests? So far import tests are always passed (if built successfully of course).

Thank you for working on that. I will try to take a look at the windows if I have some time later on the week.

Thank you! I picked up this task simply to understand how difficult it is. I have little to no experience in every aspect involved here.

Let me quickly summarize before I turn my attention to other things.
On Windows, the build fails at the configuration stage (python configure.py ...) for PyQt5. Specifically, the failure happened at run_make(), which is part of compile_test_program() that targets QtCore. In d2dfd06 I printed out the generated makefile, but I don't see anything weird there: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=74062&view=logs&jobId=171a126d-c574-5c8c-1269-ff3b989e923d&taskId=7225e372-3a5e-5b11-c153-719d50bbeacd&lineStart=567&lineEnd=1055

With --debug turned on the error becomes LINK : fatal error LNK1104: cannot open file 'Qt5Core_condad.lib' instead of cannot open shell32.lib.

I cleaned up debug prints and suppressed Windows build in 88cba5b. For Linux and Mac OS X this PR is ready to go.

Issue is that qt=5.12 was built on @msarahan's computer and it has a newer SDK than what azure has. Qt somehow saves the paths and this issue pops up.

@isuruf Oh OK do you want me to turn on the Windows build, or you can take over from here?

Do we need a build against an older SDK?  (the one on Azure, presumably?)

@msarahan, yes. we need to build qt against >=10.0.16299.0,<=10.0.17763.0

OK, I'll rebuild it when I get home tonight.

@msarahan, @mingwandroid said he might do it today as well.

Great, I'll coordinate with him on it.

@isuruf I don't think you can restore py27...It's a bug on PyQt, not on Qt, so a newer Qt (I see 5.12.5 was just built!) isn't helping.

Issue is that qt=5.12 was built on @msarahan's computer and it has a newer SDK than what azure has. Qt somehow saves the paths and this issue pops up.

I think you're right, @isuruf! Windows build is still running on Azure without failing 🎉

Try nmake throughout in Win builds. I found that jom is often unable to find the files to be copied over.

cool, it builds!

Thank you very much @isuruf for finishing the PR!


ValueError: Could not find CIFs

I have been setting up the alphafold2 along with Docker.  When I came to step 8 for test, I put the command in .sh file and run, but the following errors occurs:
~/bin/alphafold$ cat run_alphafold.sh
python3 docker/run_docker.py \
  --fasta_paths=fastafiles/test_protein.fasta \
  --max_template_date=2023-12-30 \
  --data_dir=/data/disk4/alphafoldDB \
  --output_dir=~/project/3-alphafold2-output

~/bin/alphafold $ sh run_alphafold.sh
I0313 20:59:48.014863<PHONE_NUMBER>93024 run_docker.py:116] Mounting /home/wuqy/bin/alphafold/fastafiles -> /mnt/fasta_path_0
I0313 20:59:48.223854<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB/uniref90 -> /mnt/uniref90_database_path
I0313 20:59:48.429889<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB/mgnify -> /mnt/mgnify_database_path
I0313 20:59:48.430127<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB -> /mnt/data_dir
I0313 20:59:48.432076<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB/pdb_mmcif -> /mnt/template_mmcif_dir
I0313 20:59:48.433026<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB/pdb_mmcif -> /mnt/obsolete_pdbs_path
I0313 20:59:48.434590<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB/pdb70 -> /mnt/pdb70_database_path
I0313 20:59:48.436120<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB/uniref30 -> /mnt/uniref30_database_path
I0313 20:59:48.437566<PHONE_NUMBER>93024 run_docker.py:116] Mounting /data/disk4/alphafoldDB/bfd -> /mnt/bfd_database_path
I0313 20:59:51.911179<PHONE_NUMBER>93024 run_docker.py:258] E0313 12:59:51.910264<PHONE_NUMBER>91360 templates.py:838] Could not find CIFs in /mnt/template_mmcif_dir/mmcif_files

I0313 20:59:51.911416<PHONE_NUMBER>93024 run_docker.py:258] Traceback (most recent call last):
I0313 20:59:51.911603<PHONE_NUMBER>93024 run_docker.py:258] File "/app/alphafold/run_alphafold.py", line 570, in <module>
I0313 20:59:51.911774<PHONE_NUMBER>93024 run_docker.py:258] app.run(main)
I0313 20:59:51.911936<PHONE_NUMBER>93024 run_docker.py:258] File "/opt/conda/lib/python3.10/site-packages/absl/app.py", line 312, in run
I0313 20:59:51.912092<PHONE_NUMBER>93024 run_docker.py:258] _run_main(main, args)
I0313 20:59:51.912245<PHONE_NUMBER>93024 run_docker.py:258] File "/opt/conda/lib/python3.10/site-packages/absl/app.py", line 258, in _run_main
I0313 20:59:51.912397<PHONE_NUMBER>93024 run_docker.py:258] sys.exit(main(argv))
I0313 20:59:51.912575<PHONE_NUMBER>93024 run_docker.py:258] File "/app/alphafold/run_alphafold.py", line 478, in main
I0313 20:59:51.912730<PHONE_NUMBER>93024 run_docker.py:258] template_featurizer = templates.HhsearchHitFeaturizer(
I0313 20:59:51.912881<PHONE_NUMBER>93024 run_docker.py:258] File "/app/alphafold/alphafold/data/templates.py", line 839, in __init__
I0313 20:59:51.913031<PHONE_NUMBER>93024 run_docker.py:258] raise ValueError(f'Could not find CIFs in {self._mmcif_dir}')
I0313 20:59:51.913180<PHONE_NUMBER>93024 run_docker.py:258] ValueError: Could not find CIFs in /mnt/template_mmcif_dir/mmcif_files

I have also reviewed issue #160 about the Singularity approach, but I am not familiar with it. Is there a feasible solution to address this issue based on the current Docker configuration?

Does anyone have found a solution?

For my case, the problem is that the CIF files in the pdb_mmcif/raw directories are not 'flatterned' because one gz file might be damaged. My solution was just to download the mmcif dataset again. Double check that there is a pdb_mmcif/mmcif_files directory before running the docker script.

@Ruibin-Liu Thank you. I will try re-downloading the CIF files again. But it is very unstable to download this part and i believe the files might be damaged during the occasionally disconnect


Context closed in the middle of run; frequent

Fixed this by adding a shouldReuseBrowserContext flag set by default to false


Prevent code which uses recursion

def set_random_password
  password = SecureRandom.urlsafe_base64(PASSWORD_LENGTH)
  return set_random_password unless password.match(Employees::Constants::PASSWORD_REGEX)
end

def all_entries(path, entries = [])
  response = get(path)
  entries += entries_from_response(response)
  next_page_link = next_page_link(response)
  next_page_link ? all_entries(next_page_link, entries) : entries
end

Done in #111


Add plug-in for drawing and publishing a polygon.

This plug-in allows you to draw a polygon on the screen and publish it to a topic.

:+1:
Might be nice to be able to set the z-value on the polygon, although it doesn't matter for mapviz and I don't know if anybody else cares.


Usage of "const" keyword breaks older browsers and uglifyjs2

In the last update you are using the "const" keyword in a single place.
https://github.com/snaptortoise/konami-js/blob/060339e39599c0fb9f87dcaa1bc22102ec4ed3df/konami.js#L77
This breaks in older IE (< 11) and some mobile browser. It also breaks running your code through uglifyjs2 unless I run it through babel first. How about switching it back to a regular "var" instead to make life easier for everyone? :)
Edit
You are also using the shorthand form for declaring a method in one place it seems.
https://github.com/snaptortoise/konami-js/blob/060339e39599c0fb9f87dcaa1bc22102ec4ed3df/konami.js#L102
I'm not sure about browser support here but it breaks uglifyjs2 at least.

You're right. I do want to maintain backwards compatibility. I'm considering branching this project and making a more modern ES6-friendly version on Konami-JS at some point.
I'll merge the pull request.

Thanks!
Also, you could rewrite it using ES6 right now if you add a Babel workflow to transpile a version of it down to ES5 as well. That way people get to chose which version they want without the need to maintain two different branches of the project.

That's a great idea @karl-sjogren. I'll look to do that when I can. In the meantime I've made the fix you suggested here.


Integrating the adjustment in the PID example?

In the PID example, there's a line (https://github.com/mstksg/auto-examples/blob/master/src/Recursive.hs#L103):
control  <- sumFromD c0 -< adjustment

I get why one would delay a signal in the recursive setting, but I don't think a PID should integrate adjustment in the first place.
Shouldn't it just be:
...
        let control = kp * err
                       + ki * cumulativeSum
                       + kd * fromMaybe 0 changes
        response <- blackbox   -< control
    id -< response

Ah, I read the code before the readme. But I still don't get why sumFromD is the same as system . delay
currResponse <- system . delay c0 -< control


Training

Hi,
Thanks for sharing this, it's pretty useful.
Could you advise how I can add additional classification?
I would like to add and classify gambling related images.
Also I've tried to run the Train project but getting an error on following line;
var trainedModel = trainingPipeline.Fit(trainSet);
And the error message is;
System.ArgumentOutOfRangeException: 'Only one class found in the LabelAsKey column. To build a multiclass classification model, the number of classes needs to be 2 or greater Arg_ParamName_Name'
Many thanks

Hey,
Have you made any modifications to the training code at all? On line 18 of the Program.cs file, there should be this line:
var assetsPath = @"E:\NsfwSpy\Images";

This path will need to be changed to the directory of the images folder you want to train with on your local machine which will contain children folders named after each label (Hentai, Neutral, Pornography, Sexy, Gambling) and containing images of that classification type. We currently don't publicly provide our training dataset, so you will need to source your own if you want to also classify all of the labels that NsfwSpy currently can.
The "Gambling" label will then need to be added alongside the other labels and then hopefully the training process will work 🙂 Feel free to comment here if you have any further questions or need me to clarify anything further.

Is it possible for you to let me know what path you set on line 18 of the Program.cs file and screenshot the folders inside of that please?

https://github.com/NsfwSpy/NsfwSpy.NET/assets/30019998/bb1af5ad-50fc-479f-b7a5-e8d2d9da4713
Yes sure, please find attached video. Let me know if you need anything else. Thank you again

Cheers for the video, makes it much clearer. Could you try with just .jpg or .png files in those folders? I don't think that ML.NET supports .webp images for training.

aah thanks, the issue was on following function which was skipping webp files. Now I've added webp format and it looks it's processing the files.
Many thanks again.
public static IEnumerable LoadImagesFromDirectory(string folder)
{
var allowedFileExtensions = new[] { ".jpg", ".jpeg", ".png", ".webp" };
var files = Directory.GetFiles(folder, "*", searchOption: SearchOption.AllDirectories);

Glad that's working for you. I haven't seen any information regarding whether ML.NET now supports .webp files when training. I think when classifying .webp files, NsfwSpy has to convert them into a different format beforehand. Just something to be mindful of 😊


[bug]  The retry mechanism for EE and CE TBPs have started to fail. Manual re-runs are working but failing via the webhook trigger

Laveena
The retry mechanism for EE and CE TBPs have started to fail. Manual re-runs are working but failing via the webhook trigger. Can you please help check what's causing this?
https://theappsmith.slack.com/archives/C05QG0J87UL/p1730273499439579

Please add your planning poker estimate with Zenhub @ayushpahwa

. calling rerun on github action before the action is complete throws 403 forbidden error. The added 10 mins delay before calling the rerun fixed it.


[dv, flash_ctrl] Test instruction fetch into flash_ctrl

This was missed during the flash test plan.
The flash has a feature where instruction execution can be explicitly disabled / enabled.  This function is enabled / disabled by software, and can also be turned off as part of the escalation process.
We need to make sure to test this as ROM's secure boot / alert handling process is closely tied to it.

@weicaiyang
@sriyerg
@cindychip
@matutem
@alphan

@nikola-miladinovic can you update the testplan and add a test for this?

sorry this is my fault for not being more clear.
I believe @TimEwins has implemented this test in #11411
Tim can you confirm and close if I am correct?


Purpose of Google Analytics and gtag call from this tool

Hello Team,
Please confirm what this script does in index.html
**<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id={$GOOGLE_ANALYTICS_ID}"></script>**

Does it record the pageviews of
**lwc-soql-builder github site
or
the salesforce api calls**

Sorry for the late reply.
This code uses Google Analytics only for the purpose of recording page views.


Error occurred when executing FaceSwapSetupPipeline:  [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from /workspace/ComfyUI/models/insightface/models/antelopev2/1k3d68.onnx failed:Protobuf parsing failed.

Doesn't work
Followed all instructions... any suggestions? Seems like the onnx model has an issue?
Error occurred when executing FaceSwapSetupPipeline:
[ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from /workspace/ComfyUI/models/insightface/models/antelopev2/1k3d68.onnx failed:Protobuf parsing failed.
File "/workspace/ComfyUI/execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/workspace/ComfyUI/execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/workspace/ComfyUI/execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/workspace/ComfyUI/custom_nodes/comfyui-instantId-faceswap/node.py", line 199, in create_pipeline
app = FaceAnalysis(name="antelopev2",
File "/workspace/venv/lib/python3.10/site-packages/insightface/app/face_analysis.py", line 31, in init
model = model_zoo.get_model(onnx_file, **kwargs)
File "/workspace/venv/lib/python3.10/site-packages/insightface/model_zoo/model_zoo.py", line 96, in get_model
model = router.get_model(providers=providers, provider_options=provider_options)
File "/workspace/venv/lib/python3.10/site-packages/insightface/model_zoo/model_zoo.py", line 40, in get_model
session = PickableInferenceSession(self.onnx_file, **kwargs)
File "/workspace/venv/lib/python3.10/site-packages/insightface/model_zoo/model_zoo.py", line 25, in init
super().init(model_path, **kwargs)
File "/workspace/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/workspace/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 472, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)

I fixed it by replacing the .onnx files with the ones found here instead:
https://huggingface.co/menglaoda/_insightface/tree/main
They are different; the checksums are different. The ones in this link I pasted match the onnx files used by Reactor
The ones linked in the docs today are corrupted... cheers


Bump to composer 0.24.0 and torch 2.4

Updates to torch 2.4, as composer 0.24.0 now supports it. We pin to only torch 2.4 so cuda extension libraries are also usable.

@snarayan21 I usually do this in multiple steps. (1) pr to make the docker images and move the upper pin, (2) pr to add the tests, (3) pr to move the lower pin

And please bump to composer 0.24 separately from the torch bump

closing in favor of other PRs


Perspective view window reverts to Lock View Rotation

Godot version: 3.2.3
OS/device including version:
Manjaro Linux
KDE Plasma 5.20.2
Kernel: 5.9.3-1
64-bit
NVIDIA GeForce GTX 1070 Ti, proprietary drivers
12 x Intel Core i7-8700K CPU @ 3.70GHz
32 gb RAM
Editor is set to GLES3, with fallback to GLES2 enabled because I export to Quest, in addition to desktop.
Issue description:
I set any 3D scene window that's in Perspective mode to free rotation, and lock any in Orthogonal to locked rotation. It retained these settings for a while, now it defaults to all of my Perspective windows as locked rotation, and often even reverts back to locked when I return to a tabbed scene, not just when I open it for the first time.
This problem started to happen after I created a window layout as such: View -> 3 Viewports (alt), with the largest being Perspective with unlocked rotation, and the smaller being Orthogonal with locked rotation. It never seemed to cause a problem when I had regular 3 Viewports (not alt), with the largest, unlocked Perspective being the first view, the one on top. This makes me wonder if it numbers the windows, and now that my first windows is locked in that view, it locks all the other first windows.
If I open my project, and set my first scene to unlocked, which is just one big Perspective window, other panels that open subsequently work just fine. But as soon as I open a scene that has 3 Viewports (alt), it locks all the open Perspective window scenes.
Steps to reproduce:
ViewPort_problem.zip

Open a new scene, set its view to 3 Viewports (alt). Lock the rotation for the two smaller and unlock rotation for the largest.
Open another scene, set its view to 1 Viewport, unlock the rotation.
Switch back to the first, then again to the second.
Notice that the view is now again locked.

Minimal reproduction project:
ViewPort_problem.zip

For what it's worth, I tested the same issue on my girlfriend's system, and it's not present there. I see that I have the same GPU as you (1070 Ti), whereas my girlfriend has a different one, but this sounds like a weird reason for an issue like this to appear. At the same time, I don't have a better indicator of what causes the issue.

I also realized that for me the issue is not related to Orthogonal vs Perspective, but rather it propagates from a "lock view rotated" scenes to not "lock view rotated" scenes. That is if I have 3 scenes:

A, B with "lock view rotation" turned off
C with "lock view rotation" turned on

If I'm at the scene A, and change to scene B and back, the lock view rotation stays off, but if I moved to C and back, the lock view rotation for A will get turned on.
I also observed that despite the lock view rotation being on (and being able to see it being ticked in the options menu), the "View Rotation Locked" bar at the bottom doesn't appear when the lock view is caused by coming from another scene which has lock view rotation on.

I looked through the code a bit and it seems to me that this property is set here and restored over here. Could someone confirm if I'm reading this right that this state is stored and restored across scenes, and if so, if there is a separate mechanism that would allow storing state only for a particular scene (which would make more sense in this case)?
Furthermore, am I reading it right that right now only the "locked-in" state gets stored whereas if the view is not locked, it won't get stored (which would confirm my observations above)?
@kyledayton who implemented this

I think I got to the bottom of this. The lock view rotation is stored as part of the state in Node3DEditorViewport, which is stored as part of the Node3DEditor's state, which is stored as Node3DEditorPlugin's state. The state for different plugins is indexed by the name of the plugin, which, in the case of Node3DEditorPlugin is just "3D", causing the state for one scene to leak to a different one...
I presume it should be possible to fix by attaching the scene name to the "lock_rotation" key in the state (and make sure that it doesn't get lost when moving across multiple scenes) but what bothers me is why this issue doesn't appear with other viewport properties like snapping, which are handled using the same state?


[SPARK-27577][MLlib] Correct thresholds downsampled in BinaryClassificationMetrics

What changes were proposed in this pull request?
Choose the last record in chunks when calculating metrics with downsampling in BinaryClassificationMetrics.
How was this patch tested?
A new unit test is added to verify thresholds from downsampled records.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@srowen Could you please have a look at this pull request? Thanks a lot!

This doesn't look like a bug. I can't understand the argument in the JIRA why the last vs first element of a bin is more representative. Both are approximations.

@srowen Yes, both are approximations. But it has less error if we choose the last element in each chunk as the threshold.
And the essential problem is that, the so-called "downsampling" is not real sampling. The code behind calculates AUC based on statistics (like TP, NP, TF, NF) of all elements.
counts.mapPartitions(_.grouped(grouping.toInt).map { pairs =>
  // The score of the combined point will be just the first one's score
  val firstScore = pairs.head._1
  // The point will contain all counts in this chunk
  val agg = new BinaryLabelCounter()
  pairs.foreach(pair => agg += pair._2)
  (firstScore, agg)
})

You can see, counters (BinaryLabelCounter) of all elements are merged into one instead of return the first element directly.
Thus, from the definition of threshold, the score of the last element (which is the minimal one) is the right threshold to use when inference.
In online systems, we need choose the right threshold to predict whether an instance is positive (score>=threshold) or negative (score<threshold).

I still don't see the argument that the first or last is better. They are simply the endpoints of the range of scores within the bin. As the number of bins increases, the range is smaller. If you are worried about this difference, you need more bins. Your argument cuts two ways: having a slightly higher threshold than desired can cause as many problems as slightly negative.
What would be possibly better here is to compute the score of a bin as a weighted average of its elements. That would be OK though you'd have to change many tests. I think the current implementation is designed to match scikit (?)

@srowen Get your point!
Actually, if we choose score of the last element in each chunk as threshold, the calculated Recall, Precision, FMeasure on each threshold are exactly the same as those when no sampling (numBins=0).
In other words, they are accurate metrics. The only difference is the count of thresholds when printing precision/recall/f1 curve compared to downsampling.
Thus, why not return the correct metrics of full data set but approximate values?

Ah OK I agree with you, I see the argument now. It comes from the fact that the scores are sorted descending. The score of each bin is currently its maximum, not minimum. The precision / recall for each bin is calculated as if all of the instances in the bin were classified as positive. This only makes sense if the score is the minimum.
You might mention something to this effect in the comment in the code.
Also I think this may change some test results; let's see.

Test build #4776 has started for PR 24470 at commit fa2eae6.

Test build #4776 has finished for PR 24470 at commit fa2eae6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@srowen Great thanks for your patience!
I have added explanation in code comments at BinaryClassificationMetrics.scala. Do these following words match your expectations?
counts.mapPartitions(_.grouped(grouping.toInt).map { pairs =>
  // The score of the combined point will be just the last one's score, which is also
  // the minimal in each chunk since all scores are already sorted in descending.
  val lastScore = pairs.last._1
  // The combined point will contain all counts in this chunk. Thus, calculated
  // metrics (like precision, recall, etc.) on its score (or so-called threshold) are
  // the same as those without sampling.
  val agg = new BinaryLabelCounter()
  pairs.foreach(pair => agg += pair._2)
  (lastScore, agg)
})

Besides, I have scanned all unit tests and class references in the Spark code repository. None of them uses numBins but one unit test BinaryClassificationMetricsSuite, which only tests the ROC curve without threshold. Thus, it is safe to merge this pull request.

@srowen Execuse me, is there anything I should do before merging this pull request? Thanks a lot!

No, I leave these open for a day or two to make sure there aren't more comments.

Merged to master/2.4/2.3


Fix MSB3202 Warning

There's no Stratis.Bitcoin.Common project.
Note: don't merge blindly, I am not sure what that project was intended to do.

Why does the check fails?

Closing for https://github.com/stratisproject/Breeze/pull/200


Generate summary for reference pages

Is your feature request related to a problem? Please describe.
There isn't a summary for the reference docs, making it hard to walk through different APIs if you're unfamiliar with what's available
Describe the solution you'd like
An automatically generated summary for /reference/ for all namespaces, and potentially also a summary per namespace. Probably do it in checkdocs (somewhere an AST is available so it can just take all the help links)
Describe alternatives you've considered
n/a
Additional context
Also useful as the built summar[y/ies] can be parsed easily for use in a sitemap

We added this for Arcade.


Add CSRF support to uploader Component

References

Fixes #1023
Relates to #977

Description
As noted in #1023, after merging PR #977 to add CSRF protection, all file uploads are failing with a 403 Forbidden: Access is denied. Invalid CSRF token. error.
The reason for this error is that our current upload library (ng2-file-upload) doesn't currently support Angular Http Interceptors (like our xsrf.interceptor.ts which manages CSRF/XSRF tokens for all parts of our Angular app).  This is a known issue in ng2-file-upload for many years and described in more detail at https://github.com/valor-software/ng2-file-upload/issues/950
I've solved this issue in our Uploader component by temporarily duplicating much of the logic of our xsrf.interceptor.ts into the uploader.component.ts.  This includes the following logic:

Added logic to ensure uploader.component.ts sends the XSRF token in every upload request (same logic as  xsrf.interceptor.ts)
Added logic to uploader.component.ts to check for changes to XSRF token & ensure it gets saved to a cookie (same logic as  xsrf.interceptor.ts)
Updated uploader.component.ts to ensure it always uses withCredentials = true (as this is needed to ensure it passes back cookies related to XSRF/CSRF). (also same logic as  xsrf.interceptor.ts)

Instructions for Reviewers
List of changes in this PR:

Test that uploading files now works.  For example, start a new submission and drag & drop a file
Test that an action after an upload also works (this tests that we are getting the new CSRF token after a successful upload).  For example, delete the uploaded file from the item.

@artlowel : I've flagged this as high priority, as this is getting in the way of all uploads to DSpace 7.  If you (or someone on your team) can find time to look at this on Monday, I'd appreciate it.  I'd like to find a way to get this moved forward soon.

@artlowel : Good catch.  This was a hard one to debug, as I was having trouble reproducing it when using a local REST API.  However, once I switched to using the demo REST API, I was able to find similar behavior...
Here's the way I was able to semi-reliably reproduce the error on the demo REST API:

Start a submission, make some small edits. Click "Save for later"
Go back into that same submission.
As the first action, immediately drag & drop. ~5 files (at the same time) into the submission.  In that scenario, the first file uploads successfully, while the others fail with CSRF errors.

For some strange reason, I wasn't able to see that same failure when using a local REST API.
Nonetheless, after some closer analysis of the code in ng2-file-upload library, I realized that the onBeforeUploadItem() method does NOT seem to always be called prior to uploading queued items (several in a row).  Previously, I had assumed this was called between each queued item, but it doesn't seem to be the case (it might be a bug in the library, but we can work around it for now).
So, I've pushed up a small fix to also "reset" our CSRF request header in onCompleteItem() and onErrorItem() as those are always called after each file is uploaded -- so if the upload results in a new CSRF Token, I can use that response to prepare the headers for the next file in the queue.
This small change seems to work!  I've retested with the demo REST API and local REST API and cannot reproduce those issues anymore.  Could you give this updated PR another try?

@artlowel : Just a note... based on your feedback, I've also created a small PR on the REST API to change how frequently the CSRF Token is reset/refreshed.  I've noticed myself in testing this PR that it's currently reset on every request, which means sending many requests in a row (like doing multiple uploads in a row quickly) run the risk of occasional "race conditions".  I'm not yet sure if that also could be what you saw...but I think there's a definite risk to changing this token too frequently.   So, here's that PR: https://github.com/DSpace/DSpace/pull/3168/
In any case, I think this REST API PR is only loosely related to this Upload component fix.  The two PRs together should be more "stable" (less likely to see random CSRF issues), but they can be merged/reviewed separately.


Avoid the change of sys.path to load python modules

We should discuss here how to avoid this kind of things we have right now in the code:
sys.path.insert(0, '../')

We're using this in news.py, for example, to load the utils module that is located in the parent folder.
I think should be avoided for two reason:

It breaks the linter since it doesn't resolve the path
For someone it's a bad practice and for someone else it's not. But honestly if we could find another simpler way, it would be good.

I know of no other simpler way...  This is really more trouble than it is worth to change it.
When a project is small, it is best to keep all files in the same directory.  If you really want a multilayer package and you want intra-package references you must always run your code as a package: python -m UnivaqInfromaticaBot <option>
See: https://docs.python.org/3/tutorial/modules.html#intra-package-references and http://www.dabeaz.com/modulepackage/

Well if so, I'd keep things like this.


Redesign the app

After we have a working MVP it would be a good time to stop and think what we want the UI/UX of the app to be.
Take into account:

Configuring importers

YNAB setup
Google sheets setup

Configuring exporters
Global configuration (like show browser)
Import and export progress indication
Category management  and rules for automatically assigning transactions to categories
Report after imporn export finishes

@brafdlog There are no action items here, I think it can be close. See #143


Improve deliver_scheduled_messages code path significantly

There's one major bug fix here, which is that we weren't sending an event when a scheduled message actually gets delivered, but this also adds some general code path hardening.
@amanagr @mateuszmandera @sahil839 it'd be great if one of you can review this and then push some backend tests; my guess is it's going to fail CI due to coverage.

This looks reasonable to me -- we're not dropping any state, so we can look at the table if there's an exception and potentially figure out why it failed.  This is going to generate exceptions for things like "scheduled send to a user who is now inactive" which we ideally want to surface to the user, not the admin.
As discussed elsewhere, we should have a failed column rather than lying about delivered, and it would be great to have a column which links the scheduled message to the delivered one.

Pushed to fix conflict. Changes LGTM.

OK I've pushed an updated version that has the full set of logic changes that we want. It unfortunately lacks tests, which caused me to notice that our test-backend full coverage enforcement logic was not running on zerver/actions/* -- I'll address that separately.
I'll need someone to take over the work of extending the test_scheduled_messages.py code path to test the "actually deliver the scheduled message" part; while I've manually verified the logic works in various settings, it's pretty important to have actual tests.

I'll need someone to take over the work of extending the test_scheduled_messages.py code path to test the "actually deliver the scheduled message" part; while I've manually verified the logic works in various settings, it's pretty important to have actual tests.

Ok, working on it.

Lines missing coverage:
ERROR: zerver/actions/create_realm.py no longer has complete backend test coverage
  Lines missing coverage: [167, 194]

ERROR: zerver/actions/message_delete.py no longer has complete backend test coverage
  Lines missing coverage: [27]

ERROR: zerver/actions/message_edit.py no longer has complete backend test coverage
  Lines missing coverage: [925, 1098, 1166]

ERROR: zerver/actions/presence.py no longer has complete backend test coverage
  Lines missing coverage: [119]

ERROR: zerver/actions/scheduled_messages.py no longer has complete backend test coverage
  Lines missing coverage: [111]

Looks great, marked to merge once CI passes. Thanks @amanagr, I really appreciate your finishing this for me.


fix: unique name for diag artifacts for marker runs

Test run: https://github.com/splunk/splunk-add-on-for-google-cloud-platform/pull/757

:tada: This PR is included in version 4.17.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add Mime() to exclude parameters in generated string

Add a Mime() method on the ContentType to be able to get the MIME only
part without the parameters for simpler comparison against other
MIME-types in for example switch statements.
Partially closes #3

Thank you!


Can't detect MethodOptions

I am currently in the process of migrating an existing grpc api server to connect-go. Went smoothly so far, great work !
I am actually stuck at one point, in the current implementation i detect the specified MethodOptions per service and construct ACL Rules from them.
Example Proto
service HealthService {
  rpc Get(HealthServiceGetRequest) returns (HealthServiceGetResponse) {
    option (visibility) = VISIBILITY_PUBLIC;
  }
}

Then i use @jhump "github.com/jhump/protoreflect/grpcreflect" to load the ServiceDescriptors and iterate through.
This requires a grpc.Server instance which is not available anymore.
Any hints howto get access to the MethodOptions with a connect-go implementation are welcome.

Hi, @majst01. Unfortunately, the grpcreflect package is written to use the gRPC Go runtime API. So it will not be helpful when using Connect. But there is luckily a way to do what you are trying to do, now that reflection/descriptor support has been added to the core Protobuf Go runtime API.
A Connect server has no abstraction like a "server" or "service registrar". Instead it just generates HTTP handlers that you add to your own http.Server instance. So you'll have to track the registered services yourself, which for now (unfortunately) means putting hard-coded string literals for the fully-qualified service names into your program.
For a particular service, you can get the service's or methods' options via this code:
// import "google.golang.org/protobuf/reflect/protoreflect"
// import "google.golang.org/protobuf/reflect/protoregistry"
// import "google.golang.org/protobuf/types/descriptorpb"

descriptor, err := protoregistry.GlobalFiles.FindDescriptorByName("acme.user.v1.UserService")
if err != nil { return err }
userSvcDesc := descriptor.(protoreflect.ServiceDescriptor)

// inspect service options
userSvcOpts := userSvcDesc.Options().(*descriptorpb.ServiceOptions)

// inspect method options
for i := 0; i < userSvcDesc.Methods().Len(); i++ {
    methodDesc := userSvcDesc.Methods().Get(i)
    methodOpts := methodDesc.Options().(*descriptorpb.MethodOptions)
}

An improvement in the generated code for Connect would be to add exported constants with these service names. That way you don't have to hard-code the strings in your program; if the service name is changed, the program is not broken (though that is not a backwards-compatible change), and if the service is moved to a different package, there is a compile error instead of runtime error (though, again, that is not a backwards-compatible change). This would also improve the use of server reflection, which also currently requires hard-coding service names.

Hi @jhump thanks for your explanation,
meanwhile i managed to implement it somehow with the help of 	github.com/jhump/protoreflect/desc/protoparse.
With that i was not able to get the MethodOptions in in type-safe way.
I will try your proposed way if i am able to get the types of the MethodOptions this way.
I tried this:
visibilities := proto.GetExtension(methodOpts, v1.E_Visibility).([]v1.Visibility)

But it always complained that methodOpts is expected to be of type Uninterpreted, but it was ?

An improvement in the generated code for Connect would be to add exported constants with these service names.

There are already generated constants for this! No need for hard-coded string literals anywhere.

There are already generated constants for this! No need for hard-coded string literals anywhere.

Oh, nice! Should I update the examples in the README for connect-grpcreflect-go to refer to them?

The approach that @jhump suggests works and is the best approach to this problem. I'm going to close this - @majst01, please ping this issue if you're still having trouble.

in my case, i use the following code to archive it
package middleware

import (
	permissionv1 "permission/v1"
	"connectrpc.com/connect"
)

func NewUserAuthInterceptor() connect.UnaryInterceptorFunc {
	interceptor := func(next connect.UnaryFunc) connect.UnaryFunc {
		return func(
			ctx context.Context,
			req connect.AnyRequest,
		) (connect.AnyResponse, error) {
			methodDesc, ok := req.Spec().Schema.(protoreflect.MethodDescriptor)
			if !ok {
				return nil, connect.NewError(connect.CodeInternal, fmt.Errorf("expected protoreflect.MethodDescriptor got %T", req.Spec().Schema))
			}
			var adminRequired bool

			ext, ok := proto.GetExtension(methodDesc.Options(), permissionv1.E_Permission).(string)
			if ok && ext == "admin" {
				adminRequired = true
			}

			fmt.Println(adminRequired)

...

			return next(ctx, req)
		}
	}
	return interceptor
}

@solarhell, thanks for posting that. That is indeed the best way to do this now.
The Schema field that your code is using was introduced in v1.13.0 (about a year after this issue was closed). When using protoc-gen-connect-go (v1.13.0 or later), the Schema will be populated with a protoreflect.MethodDescriptor that describes the method being invoked.

@jhump I can help add this complete example to the examples repo, as this usage is very common.


wagtailadmin.views.pages.set_page_position to work with specific pages

Fixes #3469

Closing as per https://github.com/wagtail/wagtail/issues/3469#issuecomment-288374608.


Constant name "screen_count_value" doesn't conform to UPPER_CASE naming style

https://github.com/lbonanomi/gRist/blob/27e8bad1df8ab5380e5a67697cb91b06aafed42d/grist.py#L114

Issues for naming style are now ignored.


How to set the color value of color_frame?

Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

Consider checking out SDK examples.
Have you looked in our documentations?
Is you question a frequently asked one?
Try searching our GitHub Issues (open and closed) for a similar issue.

All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

Required Info

Camera Model
D400

Operating System & Version
Linux (Ubuntu 20)

Language
python

Issue Description
I got the rgb image from the color_frame.get_data(),

and I let it go through the neural network to get a mask

I can see the color of the mask on pyglet, Probably because rgb image is used

But when I export the ply file, I get the color of the original image instead of the mask, because the pointcloud uses color_ frame

So, how do I assign the mask to color_frame so that the ply file is the color of the mask? Or whether there are other ways to achieve this effect?

Hi, the only Python pointcloud reference that I could find about defining a mask in a program that uses the export_to_ply
ply export instruction is in the SDK example program opencv_pointcloud_viewer.py
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_pointcloud_viewer.py#L255-L258

It seems that we can't get the ply file of mask color. I tried to add m = m & mask after these lines, but the displayed content did not change. What should I do?
Alternatively, if I change the color to mask before this line, I will get the mask color display. But when I press "E" to save the ply file, I get the original color instead of the mask color

Does the mask get saved to ply if you incorporate your mask change into the 'm' instruction?
m = im & jm & mask
Another approach may to to give the original 'm; a different name and then apply your mask to the output of the renamed 'm'.  for example, if you renamed 'm' to 'm_original':
im = (i >= 0) & (i < h)
jm = (j >= 0) & (j < w)
m_original = im & jm
m = m_original & mask

Do you require further assistance with this case, please?  Thanks!

Apologies for the delay in responding further.The suggestions you provided do not solve the problem. I have no other solutions.

I do not have any further references to suggest either other than looking at how masking is handled with OpenCV code in the rs-grabcuts example program.
https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/grabcuts

Was the advice in the comment above helpful, please?  Thanks!

Case closed due to no further comments received.


Invalid YAML generated? (!omap vs !!omap)

My issue is described in this SO question that I posted.
Essentially it appears like Psych is emitting !omap instead of !!omap when generating YAML. This causes problems when trying to parse the resulting YAML using standards-compliant YAML parsers.
In particular, check out this line in Psych.

Ya, we should change this.  Emitting !omap is due to historical issues, specifically that's what Syck would emit (even though it's not correct).  It's still valid YAML, and you should be able to work around it by teaching your YAML parser that !omap is the same as !!omap.
If you send a PR, I'll merge it, or I'll fix this when I get some time.


Gateway does not properly support SignalR

Steps to reproduce

Create a container that exposes a SignalR hub.
Create a (SPA) frontend container that hosts a web page which connects to it.
Create a Gateway that exposes the SignalR hub and the Frontend.
Example yaml:

resource gateway 'Applications.Core/gateways@2023-10-01-preview' = {
  name: 'gateway'
  properties: {
    application: application
    environment: environment
    routes: [
      {
        path: '/api' //REST API
        destination: 'http://${api.name}:${api.properties.container.ports.web.containerPort}'
      }
      {
        path: '/dispatchhub' //SignalR Hub
        destination: 'http://${api.name}:${api.properties.container.ports.web.containerPort}'
      }
      {
        path: '/' //frontend index.html
        destination: 'http://${frontend.name}:${frontend.properties.container.ports.web.containerPort}'
      }
    ]
  }
}

Observed behavior

After 15 seconds, the server returns HTTP Status Code 504.

504 Gateway Timeout (from service worker)

The body will be: upstream request timeout
The server Response header will be: envoy
SignalR connection is lost

Desired behavior

I'd like SignalR to keep working indefinitely

Workaround

Connecting directly to the application container works, but is not great from a CORS perspective.

rad Version
RELEASE   VERSION   BICEP     COMMIT
0.33.0    v0.33.0   0.33.0    6ffdc9980420c00f7af35b024dd0e62e598efef2

Operating system
Windows 11 x64
Additional context

Envoy does support wss
Contour does so as well, patching the HTTPProxy helps, but is not the way to go imo

Would you like to support us?

[ ] Yes, I would like to support you

AB#12363

Note from the triage team: we don't think there should be limitations for applications moving to Radius, seems like something about the way we are using Contour is causing this issue and we should fix that.

Adding a very short design document: https://gist.github.com/ytimocin/ada142d271a984e9e88f9249c82c82e6.

Hi @loekd! If you are using edge version of Radius, could you try adding enableWebsockets: true to the route(s) you need and try again?


Example is incomplete

If you compile and run example.cpp as-is, it will say "Could not find a plan!" That's because the example is incomplete, but it's a simple fix.
Before calling planner.plan in main, you need to set the initial state of both the axe and wood.
...

state.has_axe = false;
state.has_wood = false;

auto len = planner.plan(state, goal, actions.data(), actions.size(), path, max_path_len);

...

When running the example after that, you will see:
Grabbing my axe!
Cutting wood!

I would not say the example is incomplete. The issue is that most state variables are not correctly setup during initialization phase. But @ProxyProdigy you're right, correctly setting an initial value will result in a successfully running example across most C++ compilers.
I did a similar fix for Visual Studio. Additionally, I assume the example will run fine with a release build as most compilers initialize the bool variables with false.

Thanks for the pointer. I added a clean init of the state: https://github.com/cvra/goap-cpp/commit/bc6d43d199adde3df055faef6a543fc6c9b5c215


listExpenditure

View the List of Expenditure

Pull Request Test Coverage Report for Build 127

0 of 5 (0.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.08%) to 50.336%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/main/java/seedu/address/logic/parser/AddressBookParser.java
0
1
0.0%

src/main/java/seedu/address/logic/commands/ListExpenditureCommand.java
0
4
0.0%

Totals

Change from base Build 126:
-0.08%

Covered Lines:
1272

Relevant Lines:
2527

💛  - Coveralls


Tasks with dependsOn and topological sort of task start order

The current way of defining lifecycle tasks is simple and works well for single or two frameworks or libraries adding tasks.
It gets complex if many libraries need to depend on some specific one thing, that some other library provides.
We could offer:
// Module X
    lifecycle.register(
        label: "service-y",
        dependsOn: "database", // ["database", "http-server"]
        start: .async { ... },
        shutdown: .sync { }
    )

// Module Y
    lifecycle.register(
        label: "service-y",
        dependsOn: "database", // ["database", "http-server"]
        start: .async { ... },
        shutdown: .sync { }
    )

// Main
// ModuleY(lifecycle)

    lifecycle.register(
        label: "database",
        start: .async { ... },
        shutdown: .sync { }
    )
ModuleX(lifecycle)
// ModuleY(lifecycle)

lifecycle.start()

^ it does not matter where I start ModuleY in "source order" and the dependencies will always be run in a predictable order.
and perform a Topological sort for starting the tasks in the right order.
This allows one library to start the "database" and other modules of a project to say i depend on "database" without having to specify more -- if the resolver notices there is no database task defined if can bail out. Otherwise, the order is made such that all tasks that depend on the "database" are spawned after it is ready.

With structured concurrency this would be expressed differently I think, going to close.


BIN:65 Adopt database merge tool for MOB->SWI migration

@levon-vardevanyan-solarwinds , @ihar-khudzeyeu can you review this PR please?