MINOR: A bugfix for GlobalKTable with different source topics.

Description of changes:
When we use GlobalKtable, we found that if the source topic(s) of the GlobalKtable changed, then the checkpoint still keeps the old topic(s) and offsets and try to continue to consume messages from the topic(s). This issue caused kafka global stream thread was in ERROR.
The exception:
java.lang.NullPointerException: null
>---at org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.update(GlobalStateUpdateTask.java:77) ~[kafka-streams-1.1.0.jar!/:na]
>---at org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:241) ~[kafka-streams-1.1.0.jar!/:na]
>---at org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:289) ~[kafka-streams-1.1.0.jar!/:na]

There is no functionality changed
Committer Checklist (excluded from commit message)

[x ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

test it.

retest this please

retest this please

@bbejeck could you take a quick look?

Thanks @guozhangwang and @bbejeck !!

Java 8, Java 11/2.12 failed, test results already cleaned out.
Java 11/2.13 passed
retest this please

retest this please

Got an error from KDK 11 and Scala 2,12,
23:00:15 kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED
23:00:34 kafka.api.DelegationTokenEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl failed, log available in /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.DelegationTokenEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl.test.stdout
23:00:34
23:00:34 kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl FAILED
23:00:34     org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 records

Checked out the code, seems a consumer cannot get a record back within certain time.
And another error from JDK `` and Scala 2.13:
00:24:57 org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > testOuterLeft[caching enabled = true] FAILED
00:24:57     java.lang.AssertionError: Condition not met within timeout 15000. Never received expected final result.
00:24:57         at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376)
00:24:57         at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:336)
00:24:57         at org.apache.kafka.streams.integration.AbstractJoinIntegrationTest.runTest(AbstractJoinIntegrationTest.java:255)
00:24:57         at org.apache.kafka.streams.integration.TableTableJoinIntegrationTest.testOuterLeft(TableTableJoinIntegrationTest.java:529)

Seems it is also related to time. It is werid that it was successfully passed in (https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/1379/ )
I could not figure out how to solve it, could you give me a hint? @bbejeck . Thank you so much!!

retest this please

@xin-au

I could not figure out how to solve it, could you give me a hint?

It's a test that is known to be flaky from time to time.  I'll make a pass on your updates soon

@bbejeck , sound great! Thank you so much!!

I think this PR should have a proper JIRA -- it's more than a MINOR fix IMHO.

This seems to be a duplicate PR to https://github.com/apache/kafka/pull/7188 -- and I am actually not sure if the proposed fix in this PR is legit. Compare https://github.com/apache/kafka/pull/7188#discussion_r312674720

@mjsax , Thanks for pointing out the PR! Yes, seems we are working to fix the same issue.
I read the comment you shared, it makes sense that throw an exception to let the caller know the checkpoint file is corrupted. I am happy either of two PRs to fix this issue.

@bbejeck , I agreed! Thanks for reviewing my PR! In this PR, I really learn a lot. I am going to close this PR.
Thanks @guozhangwang @bbejeck @mjsax !!!


Error Occurs When Using Grammar Sampling with Functionary in Batch Requests

Issue Description
When executing the command below with batch requests (e.g., using Python multithreading to request the API), an error occurs:
python server_vllm.py --model /path/to/checkpoints/functionary-medium-v3.0 --rope-scaling '{"type": "yarn", "factor": 4.0, "original_max_position_embeddings": 8192}' --rope-theta 500000.0 --enable-grammar-sampling --tensor-parallel-size 8 --port 7782 --enable-prefix-caching

Error Log
The specific error encountered is:
INFO:     [IP Address] - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/path/to/functionary/server_vllm.py", line 65, in create_chat_completion
    return await process_chat_completion
  File "/path/to/functionary/functionary/vllm_inference.py", line 347, in process_chat_completion
    async for res in result_generator:
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 800, in generate
    async for output in self._process_request
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 967, in _process_request
    raise e
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 957, in _process_request
    async for request_output in stream
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 108, in __anext__
    raise result
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 58, in _log_task_completion
    return_value = task.result()
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 651, in run_engine_loop
    has_requests_in_progress = await asyncio.wait_for
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
    return fut.result()
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 624, in engine_step
    request_outputs = await self.engine.step_async
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 326, in step_async
    ) = prompt_template.grammar_sample
  File "/path/to/functionary/functionary/prompt_template/llama3_prompt_template_v3.py", line 179, in grammar_sample
    self.update_grammar_sampling_gen_state
  File "/path/to/functionary/functionary/prompt_template/llama3_prompt_template_v3.py", line 218, in update_grammar_sampling_gen_state
    gen_state["curr_text"] = tokenizer.decode(gen_state["curr_tokens"])
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 3811, in decode
    return self._decode
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 625, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
TypeError: argument 'ids': 'NoneType' object cannot be interpreted as an integer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function
  File "/path/to/miniconda3/envs/vllm-yzm/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/path/to/functionary/server_vllm.py", line 65, in create_chat_completion
    return await process_chat_completion
  File "/path/to/functionary/functionary/vllm_inference.py", line 347, in process_chat_completion
    async for res in result_generator:
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 800, in generate
    async for output in self._process_request
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 948, in _process_request
    stream = await self.add_request
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 696, in add_request
    self.start_background_loop()
  File "/path/to/functionary/functionary/vllm_monkey_patch/async_llm_engine.py", line 560, in start_background_loop
    raise AsyncEngineDeadError
functionary.vllm_monkey_patch.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.

An input prompt for test as an example (I have met such errors in different cases and something in common is that the --enable-grammar-sampling flag is used in conjunction with batch requests.)
<|start_header_id|>system<|end_header_id|>

You are capable of executing available function(s) if required.
Only execute function(s) when absolutely necessary.
Ask for the required input to:recipient==all
Use JSON for function arguments.
Respond in this format:
>>>${recipient}
${content}
Available functions:
// Supported function definitions that should be called when necessary.
namespace functions {

// Perform a search on Wikidata.
type search_wikidata = (_: {
// The item to search. It should not be a sentence, but should be an entity's name.
query: string,
}) => any;

// Use this function at the end of the task handling process. As the user and other agents do not have access to your intermediate
steps and the solution presented in your calling of `subtask_solver`, you should write the COMPLETE final answer in the `conclusion
` parameter of this function. Include as many information from your exploration as possible.
type submit_task = (_: {
// Use around 400 words to summarize what you have done to handle this task, especially some milestones (such as writing what conte
nt to file xxx, getting what information from web xxx). Present the final answer explicitly in details. Only this conclusion will b
e shown to user, so you must write down enough detailed information that summarize all the things and information you got.
conclusion: string,
}) => any;

// Define subtask and generate its response by yourself if you want to solve subtask rather than generate thought or call other too
ls.
type subtask_solver = (_: {
// The brief description of the subtask you want to create and solve by yourself.
subtask: string,
// your detailed and self-contained response to the subtask.
solution: string,
}) => any;

} // namespace functions<|eot_id|><|start_header_id|>system<|end_header_id|>

You are WikidataAgent, and here is your profile:
WikidataAgent can search the information of entities on Wikidata.
<|eot_id|><|start_header_id|>user<|end_header_id|>

You are asked to complete the following TASK:

# Search for Information on Nedoshivina's 2010 Paper and Kuznetzov's Work on Vietnamese Specimens
## Task Inputs (including dialogues and takeaways from PREVIOUS collaboration)
[system]: The team is collaborating to solve this problem:
Where were the Vietnamese specimens described by Kuznetzov in Nedoshivina's 2010 paper eventually deposited? Just give me the city
name without abbreviations.
[WebBrowserAgent]: To achieve our goal, we need to locate the specific city where the Vietnamese specimens described by Kuznetzov i
n Nedoshivina's 2010 paper were eventually deposited. I suggest we start by gathering information about the paper itself and any re
positories or institutions mentioned in it. WikidataAgent, could you search for information on Nedoshivina's 2010 paper and Kuznetz
ov's work related to Vietnamese specimens? This might give us clues about the deposition location.
[WikidataAgent]: I will search for information on Nedoshivina's 2010 paper and Kuznetzov's work related to Vietnamese specimens on
Wikidata. This will help us gather initial clues about the deposition location.

## Task Description
Search Wikidata for information on Nedoshivina's 2010 paper and Kuznetzov's work related to Vietnamese specimens. The goal is to ga
ther initial clues about the deposition location of the Vietnamese specimens described by Kuznetzov in the paper.

<|eot_id|><|start_header_id|>user<|end_header_id|>

Now you must generate your thought and you must not call the tools in this stage. You should respond in the following json format:
```json
{
    "thought": "your thought"
}
```<|eot_id|><|start_header_id|>assistant<|end_header_id|>

>>>

It appears that the error is related to the grammar sampling functionality, as it is triggered when the --enable-grammar-sampling flag is used in conjunction with batch requests.
Steps to Reproduce
Set up the Functionary model using the command provided above.
Send multiple requests simultaneously using a batch request method, such as Python multithreading.
Observations
The error seems to be associated with the grammar_sample method in llama3_prompt_template_v3.py.
The issue arises when decoding tokens, resulting in a NoneType object being interpreted incorrectly.
Request
Could you please investigate this issue? Any guidance or potential fixes would be greatly appreciated.

It seems fine when not using grammar sampling with multi-thread requests, but the effectiveness of tool calls has decreased...

Hi @Luffyzm3D2Y, Can you provide the Inputs (list of messages and tools) that you found enabling grammar sampling gave better results?
Because from our test, enabling/disabling grammar sampling gave almost the same result?

It seems not relevent to batch request, because I observed once the same type of error when thread is set to 1.
The complete prompt in the base case is :
text = '<|start_header_id|>system<|end_header_id|>                                                                                \n                                                                                \nYou are capable of executing available function(s) if required.                                                                                \nOnly execute function(s) when absolutely necessary.                                                                                \nAsk for the required input to:recipient==all                                                                                \nUse JSON for function arguments.                                                                                \nRespond in this format:                                                                                \n>>>${recipient}                                                                                \n${content}                                                                                \nAvailable functions:                                                                                \n// Supported function definitions that should be called when necessary.\nnamespace functions {\n\n// Execute your provided code and return the terminal output. To get the result, you must explicitly print the important information (intermediate results, final results, etc.) in\n your code using `print` function.          \ntype execute_code = (_: {\n// The code to execute. If None, the code from the file specified by filename will be executed. Either code or filename must be provided.\ncode?: string,\n// The file name to save the code or where the code is stored when `code` is None. If None, a file with a randomly generated name will be created. The randomly generated file will\n be deleted after execution. The file name must be a relative path. Relative paths are relative to the working directory.                                                          \nfilename?: string,\n// The working directory for the code execution. If None, a default working directory will be used.\nwork_dir?: string,\n// The language of the code. Default is "python".\nlang?: string,\n}) => any;\n\n// Use this function at the end of the task handling process. As the user and other agents do not have access to your intermediate steps and the solution presented in your calling\n of `subtask_solver`, you should write the COMPLETE final answer in the `conclusion` parameter of this function. Include as many information from your exploration as possible.\ntype submit_task = (_: {\n// Use around 400 words to summarize what you have done to handle this task, especially some milestones (such as writing what content to file xxx, getting what information from we\nb xxx). Present the final answer explicitly in details. Only this conclusion will be shown to user, so you must write down enough detailed information that summarize all the thing\ns and information you got.                  \nconclusion: string,\n}) => any;\n\n// Define subtask and generate its response by yourself if you want to solve subtask rather than generate thought or call other tools.\ntype subtask_solver = (_: {\n// The brief description of the subtask you want to create and solve by yourself.\nsubtask: string,\n// your detailed and self-contained response to the subtask.\nsolution: string,\n}) => any;\n\n} // namespace functions<|eot_id|><|start_header_id|>system<|end_header_id|>\n\nYou are CodeExecutor, and here is your profile:\nCodeExecutor can write and execute codes to solve given questions.\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nYou are asked to complete the following TASK:\n```\n\n# Calculate ISBN-10 Check Digit for Tropicos ID of Order Helotiales\n## Task Inputs (including dialogues and takeaways from PREVIOUS collaboration)\n[system]: The team is collaborating to solve this problem:\nCompute the check digit the Tropicos ID for the Order Helotiales would have if it were an ISBN-10 number.\n[WebBrowserAgent]: We have successfully retrieved the Tropicos ID for the Order Helotiales. Now, we need to calculate the check digit using the ISBN-10 formula. The steps are as f\nollows:                                     \n1. Treat the Tropicos ID as a 9-digit number (pad with leading zeros if necessary).\n2. Calculate the check digit using the ISBN-10 formula.\n\nLet\'s discuss the best approach to proceed with this calculation. Should we assign this task to the CodeExecutor agent, or does anyone have other suggestions?\n\n## Task Description\nUsing the retrieved Tropicos ID for the Order Helotiales, calculate the check digit as if it were an ISBN-10 number. The steps are as follows:\n1. Treat the Tropicos ID as a 9-digit number (pad with leading zeros if necessary).\n2. Calculate the check digit using the ISBN-10 formula, which involves the following steps:\n   a. Multiply each of the first nine digits by its position (i.e., the first digit by 1, the second digit by 2, and so on up to the ninth digit by 9).\n   b. Sum the results of these multiplications.\n   c. Compute the modulus 11 of the sum.\n   d. If the result is 10, the check digit is \'X\'. Otherwise, the check digit is the result itself.\n3. Combine the 9-digit Tropicos ID with the calculated check digit to form the complete ISBN-10 number.\n\n```\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nNow you must generate your thought and you must not call the tools in this stage. You should respond in the following json format:\n```json\n{\n    "thought": "your thought"\n}\n```<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n>>>\n\n'

@khai-meetkai It seems not relevent to batch request, because I observed once the same type of error when thread is set to 1. The complete prompt in the bad case is :
text = '<|start_header_id|>system<|end_header_id|>                                                                                \n                                                                                \nYou are capable of executing available function(s) if required.                                                                                \nOnly execute function(s) when absolutely necessary.                                                                                \nAsk for the required input to:recipient==all                                                                                \nUse JSON for function arguments.                                                                                \nRespond in this format:                                                                                \n>>>${recipient}                                                                                \n${content}                                                                                \nAvailable functions:                                                                                \n// Supported function definitions that should be called when necessary.\nnamespace functions {\n\n// Execute your provided code and return the terminal output. To get the result, you must explicitly print the important information (intermediate results, final results, etc.) in\n your code using `print` function.          \ntype execute_code = (_: {\n// The code to execute. If None, the code from the file specified by filename will be executed. Either code or filename must be provided.\ncode?: string,\n// The file name to save the code or where the code is stored when `code` is None. If None, a file with a randomly generated name will be created. The randomly generated file will\n be deleted after execution. The file name must be a relative path. Relative paths are relative to the working directory.                                                          \nfilename?: string,\n// The working directory for the code execution. If None, a default working directory will be used.\nwork_dir?: string,\n// The language of the code. Default is "python".\nlang?: string,\n}) => any;\n\n// Use this function at the end of the task handling process. As the user and other agents do not have access to your intermediate steps and the solution presented in your calling\n of `subtask_solver`, you should write the COMPLETE final answer in the `conclusion` parameter of this function. Include as many information from your exploration as possible.\ntype submit_task = (_: {\n// Use around 400 words to summarize what you have done to handle this task, especially some milestones (such as writing what content to file xxx, getting what information from we\nb xxx). Present the final answer explicitly in details. Only this conclusion will be shown to user, so you must write down enough detailed information that summarize all the thing\ns and information you got.                  \nconclusion: string,\n}) => any;\n\n// Define subtask and generate its response by yourself if you want to solve subtask rather than generate thought or call other tools.\ntype subtask_solver = (_: {\n// The brief description of the subtask you want to create and solve by yourself.\nsubtask: string,\n// your detailed and self-contained response to the subtask.\nsolution: string,\n}) => any;\n\n} // namespace functions<|eot_id|><|start_header_id|>system<|end_header_id|>\n\nYou are CodeExecutor, and here is your profile:\nCodeExecutor can write and execute codes to solve given questions.\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nYou are asked to complete the following TASK:\n```\n\n# Calculate ISBN-10 Check Digit for Tropicos ID of Order Helotiales\n## Task Inputs (including dialogues and takeaways from PREVIOUS collaboration)\n[system]: The team is collaborating to solve this problem:\nCompute the check digit the Tropicos ID for the Order Helotiales would have if it were an ISBN-10 number.\n[WebBrowserAgent]: We have successfully retrieved the Tropicos ID for the Order Helotiales. Now, we need to calculate the check digit using the ISBN-10 formula. The steps are as f\nollows:                                     \n1. Treat the Tropicos ID as a 9-digit number (pad with leading zeros if necessary).\n2. Calculate the check digit using the ISBN-10 formula.\n\nLet\'s discuss the best approach to proceed with this calculation. Should we assign this task to the CodeExecutor agent, or does anyone have other suggestions?\n\n## Task Description\nUsing the retrieved Tropicos ID for the Order Helotiales, calculate the check digit as if it were an ISBN-10 number. The steps are as follows:\n1. Treat the Tropicos ID as a 9-digit number (pad with leading zeros if necessary).\n2. Calculate the check digit using the ISBN-10 formula, which involves the following steps:\n   a. Multiply each of the first nine digits by its position (i.e., the first digit by 1, the second digit by 2, and so on up to the ninth digit by 9).\n   b. Sum the results of these multiplications.\n   c. Compute the modulus 11 of the sum.\n   d. If the result is 10, the check digit is \'X\'. Otherwise, the check digit is the result itself.\n3. Combine the 9-digit Tropicos ID with the calculated check digit to form the complete ISBN-10 number.\n\n```\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nNow you must generate your thought and you must not call the tools in this stage. You should respond in the following json format:\n```json\n{\n    "thought": "your thought"\n}\n```<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n>>>\n\n'

I am sure this prompt could easily reproduce the reported error. It seems to be relevant to some special tokens.

Can you provide the Inputs (list of messages and tools) that you found enabling grammar sampling gave better results?

In our studies, comparing functionary and GPT* models was not our focus, and it's also challenging to pick out cases from logs where using grammar sampling yielded better results than not using it. However, we did observe that not using grammar sampling can decrease tool call accuracy, sometimes necessitating retries. If you're interested in observing this effect, I'd recommend conducting experiments on more complex agent benchmarks, such as the GAIA benchmark.

@khai-meetkai
https://github.com/MeetKai/functionary/blob/1e5019050d8cdd569f7ac394787bc6d2b0d344a1/functionary/prompt_template/llama3_prompt_template_v3.py#L139C9-L173C30
I have found where the bug occurred in this specific case. In the function grammar_sample(), the variable grammar_sampled_token_idcould be still None after grammar sampling in some condition, and it will lead to some None values in sequentially updated gen_state["curr_tokens"], and finally cause the server to crash.

@khai-meetkai It seems not relevent to batch request, because I observed once the same type of error when thread is set to 1. The complete prompt in the bad case is :
text = '<|start_header_id|>system<|end_header_id|>                                                                                \n                                                                                \nYou are capable of executing available function(s) if required.                                                                                \nOnly execute function(s) when absolutely necessary.                                                                                \nAsk for the required input to:recipient==all                                                                                \nUse JSON for function arguments.                                                                                \nRespond in this format:                                                                                \n>>>${recipient}                                                                                \n${content}                                                                                \nAvailable functions:                                                                                \n// Supported function definitions that should be called when necessary.\nnamespace functions {\n\n// Execute your provided code and return the terminal output. To get the result, you must explicitly print the important information (intermediate results, final results, etc.) in\n your code using `print` function.          \ntype execute_code = (_: {\n// The code to execute. If None, the code from the file specified by filename will be executed. Either code or filename must be provided.\ncode?: string,\n// The file name to save the code or where the code is stored when `code` is None. If None, a file with a randomly generated name will be created. The randomly generated file will\n be deleted after execution. The file name must be a relative path. Relative paths are relative to the working directory.                                                          \nfilename?: string,\n// The working directory for the code execution. If None, a default working directory will be used.\nwork_dir?: string,\n// The language of the code. Default is "python".\nlang?: string,\n}) => any;\n\n// Use this function at the end of the task handling process. As the user and other agents do not have access to your intermediate steps and the solution presented in your calling\n of `subtask_solver`, you should write the COMPLETE final answer in the `conclusion` parameter of this function. Include as many information from your exploration as possible.\ntype submit_task = (_: {\n// Use around 400 words to summarize what you have done to handle this task, especially some milestones (such as writing what content to file xxx, getting what information from we\nb xxx). Present the final answer explicitly in details. Only this conclusion will be shown to user, so you must write down enough detailed information that summarize all the thing\ns and information you got.                  \nconclusion: string,\n}) => any;\n\n// Define subtask and generate its response by yourself if you want to solve subtask rather than generate thought or call other tools.\ntype subtask_solver = (_: {\n// The brief description of the subtask you want to create and solve by yourself.\nsubtask: string,\n// your detailed and self-contained response to the subtask.\nsolution: string,\n}) => any;\n\n} // namespace functions<|eot_id|><|start_header_id|>system<|end_header_id|>\n\nYou are CodeExecutor, and here is your profile:\nCodeExecutor can write and execute codes to solve given questions.\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nYou are asked to complete the following TASK:\n```\n\n# Calculate ISBN-10 Check Digit for Tropicos ID of Order Helotiales\n## Task Inputs (including dialogues and takeaways from PREVIOUS collaboration)\n[system]: The team is collaborating to solve this problem:\nCompute the check digit the Tropicos ID for the Order Helotiales would have if it were an ISBN-10 number.\n[WebBrowserAgent]: We have successfully retrieved the Tropicos ID for the Order Helotiales. Now, we need to calculate the check digit using the ISBN-10 formula. The steps are as f\nollows:                                     \n1. Treat the Tropicos ID as a 9-digit number (pad with leading zeros if necessary).\n2. Calculate the check digit using the ISBN-10 formula.\n\nLet\'s discuss the best approach to proceed with this calculation. Should we assign this task to the CodeExecutor agent, or does anyone have other suggestions?\n\n## Task Description\nUsing the retrieved Tropicos ID for the Order Helotiales, calculate the check digit as if it were an ISBN-10 number. The steps are as follows:\n1. Treat the Tropicos ID as a 9-digit number (pad with leading zeros if necessary).\n2. Calculate the check digit using the ISBN-10 formula, which involves the following steps:\n   a. Multiply each of the first nine digits by its position (i.e., the first digit by 1, the second digit by 2, and so on up to the ninth digit by 9).\n   b. Sum the results of these multiplications.\n   c. Compute the modulus 11 of the sum.\n   d. If the result is 10, the check digit is \'X\'. Otherwise, the check digit is the result itself.\n3. Combine the 9-digit Tropicos ID with the calculated check digit to form the complete ISBN-10 number.\n\n```\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nNow you must generate your thought and you must not call the tools in this stage. You should respond in the following json format:\n```json\n{\n    "thought": "your thought"\n}\n```<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n>>>\n\n'

I am sure this prompt could easily reproduce the reported error. It seems to be relevant to some special tokens.

@khai-meetkai
Hey, I just quickly patched this bug, and it seems to have fixed the issue with grammar sampling.
Although the idea and implementation are a bit rough, at least the program won't crash because of this error anymore. Here, I've posted the code snippet, hoping it might be helpful to you.
idea: Try including the token ids of options in grammar sampling into delta_token_ids to make sure when grammar sampling the variable grammar_sampled_token_id  could be set not None.
implement:
step 1: increase the length of logprobs from 200 in default to 10000. (functionary/vllm_inference.py)
if enable_grammar_sampling is False:
        logprobs = None
    else:
        logprobs = 10000

step 2: encode options and include them into the final token id pools for grammar sampling. (functionary/prompt_template/llama3_prompt_template_v3.py)
if grammar_sampled_token_id is None:
            selected_delta_token_ids = delta_token_ids[:200]
            option_token_ids = set()
            for option in options:
                ids = tokenizer.encode(option)[1:]
                for i in ids:
                    if i not in option_token_ids and i in delta_token_ids:
                        option_token_ids.add(i)
            option_token_ids=list(option_token_ids)
            option_token_ids.sort(key=lambda x: delta_token_ids.index(x))
            for option_token_id in option_token_ids:
                if option_token_id not in selected_delta_token_ids:
                    selected_delta_token_ids.append(option_token_id)
            print(f"len(selected_delta_token_ids):{len(selected_delta_token_ids)}")

            for i, sampled_token_ind in enumerate(selected_delta_token_ids):
                sampled_token = tokenizer.decode(
                    [sampled_token_ind], add_special_tokens=False
                )
                # ...

One significant drawback of this implementation is that it might substantially slow down the inference speed (though I haven't investigated the reasons).
If you have a better solution, please let me know. Thank you very much. Additionally, during the implementation, I found grammar sampling quite interesting, seeming to be based on a state transition sampling strategy. Is there any systematic documentation that explains this mechanism, especially how the current state is determined, or could you point me to the relevant code? Thank you very much.

Hi @Luffyzm3D2Y thank you so much for helping to debug this problem.
For step 1, it seemed to be a problem within vLLM. I once raised this issue providing the latency information but no one seemed to figure out why yet. Due to how fast vLLM's codebase has been evolving, I also haven't had the chance to find out what is the reason for the significant latency degradation in recent versions when logprobs = len(tokenizer.get_vocab()). Thus, our team decided on the logprobs value of 200 first to balance between latency and sampling accuracy.
For step 2, that's a good idea to force the sampler to consider the options too by adding them into the list of selected_delta_token_ids. However, I wonder how do we determine which option should appear first if we do not have access to their logprob values? Currently, I see that each option is added in the order that they are in in options, which does not factor in their logprob values.
Regarding the grammar sampling, yes it is based on a state transition strategy. We design a finite-state machine for each prompt template. The FSM chart for functionary-small-v2.5 is here. Functionary-medium-v3.0's FSM is similar to functionary-v2.4's which is presented below:

@jeffrey-fong
After I posted my code snippet, I just modify the code a little because I still observed the same error in some other cases.
I once set as you said logprobs = len(tokenizer.get_vocab()), and I also found the significant decrease of the latency which I still have not found the reasons behind. But it's really slow, like less than 1 token/s for output.
And as your second point about how do we determine which option should appear, I think in theory, if we get the whole logprobs (like logprobs = len(tokenizer.get_vocab())), and sort the token ids from options like option_token_ids.sort(key=lambda x: delta_token_ids.index(x)), the problem could  be totally solved. But the necklace is the significant decrease of latency when we get all the logprobs.


fixing integration tests

So after much trial and error (3 branches)
https://github.com/casz/GitLabApiClient/tree/fix/dockerTests
https://github.com/casz/GitLabApiClient/tree/docker
https://github.com/casz/GitLabApiClient/tree/dockerv2
I finally went GTFO on travis and switched to GitHub action for Linux CI as it behaves as what I could produce locally, so definitely a lot of tests that are not passing.
There are still some issues running Windows CI with containers that is apparent on both AppVeyor and GitHub actions.
@nmklotas This is a step forward at least to fixing the integration tests!
Basically travis seems to have issue collecting xunit's standard output and hosting the gitlab container.
fixes #44
fixes #71

Codecov Report

Merging #100 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master     #100   +/-   ##
=======================================
  Coverage   32.12%   32.12%
=======================================
  Files         131      131
  Lines        1659     1659
=======================================
  Hits          533      533
  Misses       1126     1126

Impacted Files
Coverage Δ

src/GitLabApiClient/IssuesClient.cs
31.25% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34a20ea...25f2847. Read the comment docs.

nice, travis does not work good with this

@nmklotas it became Christmas after all! 70.01% code coverage 😱

70 cov is a lot :dagger:

I'd like to rebase and merge to keep commit history.


UML-1189 - Deploy images without dev dependencies

Purpose
Build and test the service, and deploy an environment that does not contain development dependencies.
Fixes UML-1189
Approach

[x] Check docker caching behaviour on CircleCI

caches are used and available during rebuilds

[x] Explore docker build targets option

Build targets work well on the pipeline for API App
Local build times are within an appropriate range of master having changed one Dockerfile

[ ] Explore docker volume mount option

Learning
https://docs.docker.com/develop/develop-images/multistage-build/
Checklist

[ ] I have performed a self-review of my own code
[ ] I have added relevant logging with appropriate levels to my code
[ ] I have updated documentation (Confluence/GitHub wiki/tech debt doc) where relevant
[ ] I have added tests to prove my work
[ ] I have added welsh translation tags and updated translation files
[ ] I have run an accessibility tool on any pages I have made changes to and fixed any issues found
[ ] The product team have tested these changes

Definite benefits from caching on docker builds

A targetted build (which ran a composer install twice, once for dev deps and again for no dev deps but only copied in non-dev deps) completed in similar time to previous builds

It was expected at this stage to fail at unit testing as those dependencies were not available

Building a production target after a development target uses a cache, shows good performance

A local no-cache rebuild of the service with the multistage file took 31 minutes on my machine
109.63s user 141.35s system 13% cpu 31:17.49s total
Compared to master
97.44s user 124.83s system 12% cpu 28:37.88s total

Step 3/18 : RUN composer install --prefer-dist --no-dev --no-suggest --no-interaction --no-scripts --optimize-autoloader --ignore-platform-reqs
 ---> Running in 7bb9fbcb82e3
You are using the deprecated option "--no-suggest". It has no effect and will break in Composer 3.

Removing the --no-suggest option as per the warning

Without dev dependencies, the build would return only

Is this due to FILTER_VALIDATE_BOOL being recognised with dev dependencies where it should be FILTER_VALIDATE_BOOLEAN?

Without dev dependencies, the build would return only

Is this due to FILTER_VALIDATE_BOOL being recognised with dev dependencies where it should be FILTER_VALIDATE_BOOLEAN?

Improved naming of jobs on the pipeline

Improved naming of jobs on the pipeline


fix: add hash to lib chunk names

Description
Added the hash to file names of lib chunks and assets, to avoid name confliction when using dynamic imports

Update: The asset name can't be modified currently because style.css is the hard-code name for lib. Perhaps we should add an option to customize this name in the future

What is the purpose of this pull request?

[x] Bug fix
[ ] New Feature
[ ] Documentation update
[ ] Other

Before submitting the PR, please make sure you do the following

[x] Read the Contributing Guidelines.
[x] Read the Pull Request Guidelines and follow the Commit Convention.
[x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
[x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
[ ] Ideally, include relevant tests that fail without this PR but pass with it.

Would you provide a reproduction of the issue this PR is solving?

Would you provide a reproduction of the issue this PR is solving?

OK, I'll submit an issue with a reproduction later.

Submitted #8120 to provide a reproduction link


Multipart uploads fail with minio client due to mismatched etags

I was trying to use gofakes3 to test the use of the https://pkg.go.dev/github.com/minio/minio-go/v7#Client in a package. However, I've found that multipart uploads fail due to a failing test here.
The condition fails because inPart.ETag (from the xml doc) is not quoted and upPart.ETag (from the header) is.
Even though RFC7232 specifies that etags are always quoted, aws s3 docs are not clean on whether the etag should be quoted in the xml document.

Hi @tasdomas
Thank's a lot for the contribution, seems to be a relevant change. Thanks!


Reduction Operators

Hi,
Thanks for your great work.
Instead of the local aggregation operators, I am curious about the following reduction operators. In your codes, the MAX or AVG seems different from the regular max/avg-pooling along neighbors, due to certain masks involved. (and MAX operators changes in pytorch and tensorflow implementation?).
Could you please explain a bit more on them especially for the masks?
Thanks a lot.

Hi @ShiQiu0419
This an implementation difference between tensorflow and pytorch for variable input and neighborhood. For tensorflow, we calculate the subsampling indices and neighborhood indices in dataloader following KPCONV(https://github.com/HuguesTHOMAS/KPConv). For pytorch, we do subsampling and neighborhood quering as basic operators in network. So to deal with variable input and variable neighborhood, we add paddings for points and neighborhoods. These masks indicate these paddings. And in Reduction functions, these masks are used to eliminate these paddings for more accurate results.
Hope this help!
Best,
Ze

Hi @zeliu98
Thanks for your reply. Sorry that I am not farmiliar with the implementation of KPConv.
So actually, your reduction functions aim to max/avg-pool along neighbors (like regular PointNet++ did), while extra operations in your codes only deal with some specific masks in your/KPConv settings?

Hi @ShiQiu0419
I'm a little confused about what you mean.
Let me take an example. For finding neighborhoods, we use ball query (set a ball radius and find all neighbor points in this ball), but the number of neighbor points is different for each center point due to different densities. So we usually set a max neighborhood number K. For center point which has  more than K neighbors, we choose the K nearest neighbor points; while for center point which has less than K neighbors, we use some padding points to fill to K neighbor points. But in reduction functions, we don't want to use these padding points, so we set a mask to indicate them and only use the true neighbor points. That's why we set these neighborhood masks and how they are used.
Best.
Ze

Hi  @zeliu98
Many thanks for the clarification.
I was unclear about the "mask" or "padding points" before (I suppose there was no such mask in PointNet++..) Anyway, it makes sense to me now.  Really appreciate your great efforts on this work.


data types on parameter binding

Not tagged correctly in the Optic client.

Parameters should be binding correctly now.
For sem.iri values, I had to check in a fix to the REST API as well.  You should, however, be able to test bindings on Xs*Val values without needing a new server build.

Refer to issue https://github.com/marklogic/java-client-api/issues/528


Test mptcpize

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 5 (0.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-2.2%) to 45.54%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/mptcpize.c
0
5
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
-2.2%

Covered Lines:
1021

Relevant Lines:
2242

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 5 (0.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-2.2%) to 45.54%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/mptcpize.c
0
5
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
-2.2%

Covered Lines:
1021

Relevant Lines:
2242

💛  - Coveralls


Add empfangsbevollmaechtigter

Short Description

We discovered that we need more fields, namely the Empfangsvollmacht fields. This adds them.

Changes

Add fields to input
Add output xml objects
Adapt tests

Feedback

None specific

How to review this Pull Request
Link to Confluence (internal)

The duplicated code that SonarCloud finds is merely in tests, which shouldn't be a problem and doesn't have any value in changing, So I'll merge regardless once I have an approve^^


English translation fixes + Fr translation

Please look at the 2 highlighted fields with a note
Can you push a beta branch with the fixed translations and french so that I can compile and test it before finalizing the translations ?

Note: the english was really bad and hard to understand :-)
Once the menus are fixed, I will look at the help files for eng fix + fr translation
Best regards

edit: I followed the code as much as I could to do the translations, so I found many confusing menu dependencies that I am trying to fix partially by submitting the patches

Thank you for your translation.
2.28-translation branch has been pushed.
See Sentaroh/Urilities/message_translation_utilties.xlsm for more information around the common Dialog.
Best regards.

great, I will focus on finishing translation and eng fixes before looking at the other points


But the load balancer is still visible in the console(UI), and there is a note on the load balancer: This load balancer has TCP/UDP mixed configuration. Use gcloud or REST API to delete it.
And after calling gcloud command to delete it, a failure is returned: ERROR: (gcloud.compute.forwarding-rules.delete) Could not fetch resource:
Thanks!

ForwardingRulesClient.delete() is a long running method; it starts a background long running operation (that could be successful or could fail) and it cannot immediately return success or failure. We will update the library soon (within weeks) to support the long running operations, after which, const [operation] = await client.delete(...); const [response] = await operation.promise() will throw an error if the operation has failed.
I'll close this issue for now, you can follow the long running operation support progress in https://github.com/googleapis/gapic-generator-typescript/pull/1150.


setup lighthouse workflow to check the site for performance and accessibility

should be able to adapt from geniza or startwords

implemented and reviewed in PR #23


[RDBMS] Update Python SDK version for PostgreSQL flexible servers. Add support for custom iops by adding performance-tier as a parameter

Related command
az postgres flexible-server create -g testGroup -n testServer --location testLocation --performance-tier P15' will allow to add custom IOPS using performance-tier "P15" during flexbile server creation. az postgres flexible-server update --resource-group testGroup --name testserver --performance-tier P15` will allow to update the custom IOPS using performance tier "P15" for the existing flexible server.
Description

Update CLI to use latest Python SDK version.
Add support for features:

Custom IOPS during flexible server creation and update
Add support for tier, sku-name and storag-size during replica server creation

Testing Guide
Verified CLI changes manually.
History Notes
[RDBMS] az postgres flexible-server replica create: Add support for parameters like --tier, --sku-name, --storage-size during replica creation
[RDBMS] az postgres flexible-server update: Add support for custom IOPS update for flexible server using --performance-tier
[RDBMS] az postgres flexible-server replica create: Add support for parameters like --tier, --sku-name, --storage-size during replica creation

This checklist is used to make sure that common guidelines for a pull request are followed.

[X ] The PR title and description has followed the guideline in Submitting Pull Requests.

[X ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

RDBMS


Nginx is not working for me locally

Even with a local .env file, I am still seeing this problem
DEVLAKE_ENDPOINT=localhost:8080
GRAFANA_ENDPOINT=http://localhost:3002

Unable to reproduce. nginx is part of config-ui image, and should be verified using User Setup Process. I suggest you follow these step to verify:

docker pull mericodev/devlake
docker pull mericodev/lake
git checkout config-ui-githubproxy       # this branch fixed other docker-compose issues, didn't modify nginx.conf
backup your .env and cp .env.exmple to .env
docker-compose up -d
access http://localhost:4000, all api should be proxied to devlake correctly.

What is this? docker pull mericodev/devlake

I only see these

@joncodo mericode/lake it is then

This is now working for me. Once I merged your latest pr:
    volumes:
      - ./.env:/app/.env

https://github.com/merico-dev/lake/pull/748/files


Migrate to VitePress?

https://vitepress.vuejs.org/guide/what-is-vitepress

Not right now, waiting until the project gets stable.


[KFNBC]: os.environ in a nb server exposes IP addresses of all running nbs

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
If I do
import os
print(os.environ)

in a notebook server, I will get back a list of environment variables, which include IPs for all running notebooks (along with ports for connection, which are the default 443/80, but could be more sensitive if we had custom port mapping).
Expected Behavior
I should not be getting the IPs of everyone else's notebook servers
Steps To Reproduce
Spawn a notebook
in a code cell, write
import os
print(os.environ)

Workaround (if any)
No response
OpenShift Version
No response
Openshift Version
No response
What browsers are you seeing the problem on?
No response
Open Data Hub Version
No response
Relevant log output
No response

@lugi0 I wasn't aware of this behavior, any ideas why is it happening?

@samuelvl I'm not sure. Currently in RHODS what gets exposed is similar but limited to your own running nb:
KUBERNETES_PORT tcp://XXX.XX.X.X:443
KUBERNETES_PORT_443_TCP tcp://XXX.XX.X.X:443
KUBERNETES_PORT_443_TCP_ADDR XXX.XX.X.X
KUBERNETES_PORT_443_TCP_PORT 443
KUBERNETES_PORT_443_TCP_PROTO tcp
KUBERNETES_SERVICE_HOST XXX.XX.X.X
KUBERNETES_SERVICE_PORT 443
KUBERNETES_SERVICE_PORT_HTTPS 443

Now the same information (for TLS) seems to get exposed for all (running?) servers. Perhaps at spawn time something is fetching this info for all jupyter-nb-* pods in the namespace instead of narrowing it down by username?

What I see in the logs of the pod might confirm what I said earlier:
+ JUPYTER_ENABLE_WEBDAV=false
+ export JUPYTER_ENABLE_WEBDAV
+ '[' -f /var/run/secrets/kubernetes.io/serviceaccount/namespace ']'
++ echo jupyter-nb-lgiorgi-0
++ sed -e 's/^\(.*\)-[^-]*-[^-]*$/\1/'
+ DEPLOYMENT=jupyter-nb
++ cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
+ NAMESPACE=opendatahub
+ WEBDAV_REALM=opendatahub/jupyter-nb

There's a sed on the pod name which returns jupyter-nb, which is then set as the Deployment (and as the WEBDAV_REALM)
++ echo jupyter-nb-lgiorgi-0
++ sed -e 's/^\(.*\)-[^-]*-[^-]*$/\1/'
+ DEPLOYMENT=jupyter-nb

I'm not sure if this is then used anywhere else to fetch those environment variables

My best guess at the moment is that the culpript is /opt/app-root/etc/jupyter_server_config.py, but I cannot find that file in the pod

What I see in the logs of the pod might confirm what I said earlier:
+ JUPYTER_ENABLE_WEBDAV=false
+ export JUPYTER_ENABLE_WEBDAV
+ '[' -f /var/run/secrets/kubernetes.io/serviceaccount/namespace ']'
++ echo jupyter-nb-lgiorgi-0
++ sed -e 's/^\(.*\)-[^-]*-[^-]*$/\1/'
+ DEPLOYMENT=jupyter-nb
++ cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
+ NAMESPACE=opendatahub
+ WEBDAV_REALM=opendatahub/jupyter-nb

There's a sed on the pod name which returns jupyter-nb, which is then set as the Deployment (and as the WEBDAV_REALM)
++ echo jupyter-nb-lgiorgi-0
++ sed -e 's/^\(.*\)-[^-]*-[^-]*$/\1/'
+ DEPLOYMENT=jupyter-nb

I'm not sure if this is then used anywhere else to fetch those environment variables

@lugi0 This seems to be set by the minimal notebook https://github.com/thoth-station/s2i-minimal-notebook/blob/master/builder/run#L14. But I am not sure why same issue was not observed by JupyterHub.

@lugi0 @samuelvl
As given in the Kubernetes docs,  env variables related to services are shared across pods in a namespace if the service is created after the pods are deployed.
From the Note in the docs
When you have a Pod that needs to access a Service, and you are using the environment variable method to publish the port and cluster IP to the client Pods, you must create the Service before the client Pods come into existence. Otherwise, those client Pods won't have their environment variables populated.

If you only use DNS to discover the cluster IP for a Service, you don't need to worry about this ordering issue.


Quick Question

Hey there!
Came across this from the Lootr github and saw you had used polymer to make this happen. Is this production useable?
We've been looking for something like this for so long and thought it was a miracle when we stumbled across your link!

Yes, this should work fine in production :)
Of course, if you find any issues, please do report them here!


[PE-D][Tester E] UI doesn't show when 2 volunteers' name are exceedingly long

The character limit for a volunteer name is 100 characters, however when 2 volunteers have really long names and have similar name that the UI is able to show, it may be hard to differentiate the 2 volunteers by name if the user is not aware that i requires upsizing the GUI. This is the 2 commands I used.
/v new n/SuperLongVolunteerNameThatMightNotBeFullyVisible p/91234567<EMAIL_ADDRESS>d/2024-01-01.
/v new n/SuperLongVolunteerNameThatMightNotBeFullyVisible123 p/91234567<EMAIL_ADDRESS>d/2024-01-01.
I suggest maybe limiting the allowed number of characters to what the default UI is able to show.
Or add the a tip in the UG for the user to scale the GUI to see the full name if such situations where to occur.

Labels: type.FeatureFlaw severity.Low
original: muller317/ped#6

name of 100 characters is able to be displayed when VolunSync is set to fullscreen, as seen below

following the guidelines on the course website, this is not a bug.

Furthermore, as names are limited to 100 chars in length, all names should be able to be displayed in fullscreen.
Nonetheless, thank you for flagging this issue.


[cw-storage-plus] Prefix::keys method never fails

Keys are just skipped if deserializating of its key or value fails.
That's because de_fn returns Result which implements Iterator as well and, being iterated, returns nothing in case of error value
https://github.com/CosmWasm/cw-plus/blob/b2a15611d663109d1794e2827f1cf25038e5b9a6/packages/storage-plus/src/prefix.rs#L188
Is it desired behavior?

Good point. I think this arose because of backwards compatibility reasons when implementing key deserialisation on top of the existing API. I agree this would be good to change.

Thank you


Pin to triad<0.9.2

This should unblock CI until https://github.com/fugue-project/fugue/issues/526 is resolved upstream

Codecov Report

Merging #1263 (a757243) into main (7ce5ea9) will decrease coverage by 0.46%.
Report is 3 commits behind head on main.
The diff coverage is n/a.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@            Coverage Diff             @@
##             main    #1263      +/-   ##
==========================================
- Coverage   85.55%   85.09%   -0.46%
==========================================
  Files          77       77
  Lines        4257     4255       -2
  Branches      758      752       -6
==========================================
- Hits         3642     3621      -21
- Misses        446      461      +15
- Partials      169      173       +4

see 7 files with indirect coverage changes
:mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!


[Feature Request] Automatically create TLSA records when updating certificates on compatible DNS providers

References:
https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities
https://www.huque.com/bin/gen_tlsa

Do any of the current providers support TLSA records already? In any case, i think this would probably need to be implemented upstream in the lego ACME library.


feed output is still generated before nuxt generate

Hi
I am referring to https://github.com/nuxt-community/feed-module/pull/43.
I am using v.1.1.0 of @nuxtjs/feed and v2.10.1 of nuxt.
When I generate my app, it starts by running the create function and not my normal page generation.
It is also still making a feed.xml file in my static folder.
So I think this pull is not in a new release of this module? Is this correct?
If so, could you please make a new release of this module so we can all enjoy this new feature?

Yes, the release is still pending.

@manniL Thanks for your reply.
Just to understand the approval process better, what is it that is blocking a new release? A code review? Or is there something I can help with?

Resolved by https://github.com/nuxt-community/feed-module/commit/1224ecc2bff295067f10b55f73bbf5a866b39e99


Abstract modules

This is still raw. Just throwing it out there...
Would be nice to allow swapping implementations of the common reusable modules. E.g. bootique-jdbc is currently implemented with Tomcat connection pool. bootique-cayenne depends on on bootique-jdbc. Say we want to swap Tomcat for HikariCP. This should work automatically as long as (yet non-existent) bootique-hikaricp is added to the app modules.
Bootstrapping should work through META-INF/services, or perhaps by adding more metadata methods to BQModuleProvider , showing which modules our module "implements".
OSGi may provide some inspiration as well...

The first case of an abstract module is available since 0.25 in bootique-jdbc per https://github.com/bootique/bootique-jdbc/issues/48 and friends. It allows to use common JDBC  API within Bootique apps that has no real implementation that knows how to start a DataSource. Implementations are provided by other modules. Those modules would implement ManagedDataSourceFactory to load a DataSource. It is a polymorphic configuration, and is picked up automatically from YAML configurations. You can have multiple implementations in the same app, each starting a different DataSource.
All of this was done using pre-existing Bootique API. The only new abstraction is introduced to handle a special case of "default" implementation. In case of JDBC it was important for backwards compatibility to support YAML configs that do not have an explicit factory type. So a dynamic default type detector was added that operates based on explicit extension declarations in DI.  Not sure if this is an important scenario in general, but  it can be abstracted into base Bootique if needed.

Closing. We don't need a special abstraction. bootique-jdbc is now an abstract module without any new concepts in Bootique.


Smartpy compiler update

What does this PR do?

Updates SmartPy compiler to the latest version
Fixes module 3 content

@manangouhari  Please check if the compiler is working as expected now or not.

YES! PERFECTO


[Need help] How to play wav file on both android and ios?

Hello developers!
I want to play wav file on application. Here is my code.
FloatingActionButton(
        onPressed: () async {
          await controller.setNetworkDataSource(
              'https://mybucket/test_audio.wav',
              autoPlay: true);
          // controller.playOrPause();
        },
        tooltip: 'Play an audio',
        child: Icon(Icons.add),
      ),
    )

E/IJKMEDIA(29714): https://mybucket/test_audio.wav: Invalid data found when processing input

How to solve this problem?

Now, the ffmpeg of flutter_ijkplayer haven't contains wav format, I think you need compile the ijkplayer . And replace so and framework into your project.
The current compilation configuration file is here, the specific compilation method can see this document.
For other questions about compilation can go to bilibili/ijkplayer

Thanks I will try it


Export stats bug

I think there are inconsistencies in the stats output, which I'd like to investigate further, but I came across a problem with "export statistics" and so am unable to continue testing. Basically, QControl fails to save a statistics file even though it looks like everything is fine. The"export hierarchy sheet" works fine. The files are here Z:\HBP_Analytics\QControl\QControl_GroundTruthData\H_3sections_customised.json. Would you check if it's a QControl bug or if there is an error in the JSON file? I've tried closing QControl and reopening but it still won't export stats for this file. Everything looks fine otherwise.

To give you a bit of background the dataset was created as a ground truth dataset. I intend to mark up and manually count the markers for a very coarse level of the hierarchy. I started this process and couldn't get the numbers to match up and so simplified it further to make sure the mismatch wasn't due to a counting error on my part, but now I can't export stats. So not sure what's happening here.

Thanks, it was a subtle one: export worked after paging through the sections, but not before. Should be okay now.

@sharoncy could you validate if this is fixed?

This is fine. Close 03.09.21


Provide link to genesis

UI Version: 0.8.2
Description:
To make joining a testnet easier, it would be nice to enable a user to provide a link to a genesis. This way, we don't need to rebuild Voyager for each testnet.
Downside: New testnets usually need newer SDK versions, we would need to rebuild as well.

This doesn't reflect our process anymore as we pull the genesis files from the SDK repo


fix: change transaction commitment check for output notes

Closes #252
One of the conditions to check if a transaction was commited or not is whether it's output notes are also commited. With the addition of onchain notes, the output notes created by a transaction may not be tracked by the client so we need to check with the node rpc endpoint to verify the note commitments.

Given what was discussed in #252 this PR was rendered unnecessary. The previous solution (which was commented) should work.


Update docs to match updated logo and color palette

Test Plan

python scripts/generate_mkdocs.py
mkdocs serve

Superseded by #5283.


add a line or something between chromosomes for bedgraph (bw) tracks on higlass

Commit:
Steps to reproduce

Observed behavior

Expected behavior

@SooLee Could you give a little (visual) example what you're looking for? Maybe it's already supported.

Chatted with Soo yesterday: She just wanted to point out that for most end users, the chromosome grid and labels should be on by default.
To follow up with that, I talked with Chad after the standup this morning, and we agreed that it isn't something that we should try to hard-code into HiGlass, but that the client code he's writing should handle that. So, I'll close this issue, and on slack I'll ask Chad to confirm that my understanding is correct, and if it's useful, add a cross-reference from this to his work.


Expand box on big text

I'm using it with API text, and need open folder and it height increase

Hi can you share screenshot i need to see what's the problem. When you increase text size then cell height increases?


The Commit status from github actions is always set as pending

When A commit status is set by github actions on a repository, It's currently always shown as pending even though the status is set to pass
For example, the master commit status on the sitture/env-config is green but is showing as pending.

Can you assign me the issue?

Hi Haroon,
I'm looking into it and it looks like if a repo has no status checks or context it will return no statuses and be considered pending. I'm very new to this so not sure if that helps.


Missing trailers override?

I am trying to use connectrpc-es with an nginx grpc-web proxy https://nginx.org/en/docs/http/ngx_http_grpc_module.html
I have the proxy working, with the exception that connectrpc-es throws an error over "missing trailers" despite having received the message intact. I am aware that nginx should send trailers, but it would seem that they're not planning to do that https://github.com/grpc/grpc-go/issues/892.
Is there any way I can work around this so that connectrpc-es will forego throwing an error when the trailers are missing? I was hoping for something like shouldThrowOnMissingTrailers: false but I checked the docs and see no such configuration options.

Hey Gryphon, the error message is not about actual HTTP trailers. Since browsers do not expose headers to JavaScript, gRPC-web encodes the trailers that gRPC uses into the response body. Payload messages and trailers are delimited with a binary framing.
For some reason, it appears that the response body does not contain the gRPC-web trailer. We cannot ignore this case, since gRPC and gRPC-web encode errors in the trailer. Without trailers, you could call RPCs all day, and will never know that they failed :sweat_smile:
I suspect that something else is happening. Can you:

Make sure that you have set up CORS correctly, if you are calling RPCs across origins. See here for thorough documentation: https://connectrpc.com/docs/cors
Check request and response:

Request field Content-Type should be application/grpc-web (+proto or +json is fine).
Same for response field Content-Type.
For successful calls, response field Grpc-Status should be 0 or omitted, and response body should be non-empty.
For failed calls, response field Grpc-Status may be > 0 - in this case, the body should be empty.

If this does not help you resolve the issue, please feel free to share request and response details from the browsers network explorer here, and we might be able to help.

Hey Gryphon, the error message is not about actual HTTP trailers. Since browsers do not expose headers to JavaScript, gRPC-web encodes the trailers that gRPC uses into the response body. Payload messages and trailers are delimited with a binary framing.
For some reason, it appears that the response body does not contain the gRPC-web trailer. We cannot ignore this case, since gRPC and gRPC-web encode errors in the trailer. Without trailers, you could call RPCs all day, and will never know that they failed :sweat_smile:
I suspect that something else is happening. Can you:

Make sure that you have set up CORS correctly, if you are calling RPCs across origins. See here for thorough documentation: https://connectrpc.com/docs/cors
Check request and response:

Request field Content-Type should be application/grpc-web (+proto or +json is fine).
Same for response field Content-Type.
For successful calls, response field Grpc-Status should be 0 or omitted, and response body should be non-empty.
For failed calls, response field Grpc-Status may be > 0 - in this case, the body should be empty.

If this does not help you resolve the issue, please feel free to share request and response details from the browsers network explorer here, and we might be able to help.

Thanks for the quick response. CORS is setup, content-type is set correctly, the response has no gRPC errors, the only evidence of anything going wrong is the "missing trailers" error being thrown by connectrpc-es. In fact, if I set a breakpoint on the thrown error, I can even see the data payload was received, but then it throws the error anyway over the lack of trailers in the response.
In searching this error, I've found evidence online indicating that this is a quirk of the nginx implementation of grpc-web https://github.com/grpc/grpc-web/discussions/1322#discussioncomment-9367998 (also see link in my previous comment indicating that nginx does not intend to address this).
I understand that nginx really should stick to the spec, but from what I can see, the lack of trailers is not actually interfering with the receipt of a well-formed payload. This is why I ask if we might have a way of putting connectrpc-es into a more forgiving mode where it will not throw when the trailers are missing.
Also, "just use Envoy" is not an option. I can confirm that it does make this missing trailers issue go away, but for other reasons, it is not viable for our use case.

evidence online indicating that this is a quirk of the nginx implementation of grpc-web

Okay, I'm starting to see what's going on, thanks for the link to grpc-web discussion. To clarify: nginx does not implement grpc-web with the ngx_http_grpc_module. The purpose of the module is to pass gRPC requests to gRPC servers. This is not a quirk of it's grpc-web support - it doesn't support grpc-web.
While it's possible to add CORS headers and tweak the Content-Type header with nginx, the crucial part for gRPC-web - encoding trailers in the body - is missing.
If all you need is support for the happy path, the best way forward is to copy grpc-web-transport.ts into your project, and to modify it not to throw on line 227.
If we were to remove this check, users could easily get into a situation where everything seems to work, but the server is broken in a way that prevents the client from reporting an error. I hope you understand that this is not a good option.

Of course not. The check should absolutely stay. My suggestion was opt-in configuration to bypass it. I would understand hesitance to do this because it seems a specific implementation (nginx) is the problem, even if it is only one of a few options serving this need.

I had the same issue but i did not using Nginx. But i was getting missing trailer with no reason! That was interesting to me why is this happening. And after hours of researching i found that createGrpcWebTransport raises the error. And i migrated to createConnectTransport and problem fixed ^-^.

We will not add an option to bypass the check. It will look like a solution, but it will completely discard all errors reported by the server, and we cannot guarantee that calls might hang or return invalid data (e.g. a unary call with an empty body without trailer data).


update github owner

Need to update repository owner to make build work correctly

Thanks for your assistance on this.
I did not noticed the issue, I'm still building from the previous repo.
Will update everything in the coming months and only keep this one.
Kind regards


Handle YAML frontmatter

If the document starts with a block of valid YAML delimited by ---, present it as a YAML code cell.
Additionally, parse the frontmatter and attach it as document metadata, and keep it up to date by tracking document changes.

Why do you want the yaml frontmatter as document metadata?

I am working on a couple of notebook kernels. I don't intend to use them outside of VSCode, and I like the markdown notebook format, so I want to use this extension instead of writing my own format. It would be useful to be able to store metadata, and doing in YAML frontmatter is the most straightforward way to do that.
It's been a while since I was actively working on this, but IIRC, I want to configure whether outputs are saved. Due to the nature of markdown, the most straightforward way to save outputs is as an embedded XML blob. Users who want to use the notebook as a regular markdown document as well may not want such XML blobs. If I can derive metadata from the YAML frontmatter, I can use that to configure features such as saving cell output. One of my kernels aims to provide a friendly way to query databases; a switch enabling saved outputs could be very useful in that case.

Interesting usecase, and if you want to use markdown notebooks for something I'd like to enable that in this extension rather than have other slightly different markdown notebook extensions. But I don't really understand why markdown files would have outputs, or why you would use markdown files to query databases. Wouldn't you rather use a different format for that?

why you would use markdown files to query databases.

I wrote this kernel, which supports SQL queries. The following is an incomplete example of what the Markdown could look like:

Connect to a database
notebook.Connect("my-connection", "<type>", "<connection string>")

Query the database
SELECT * FROM MyTable

Here is a screenshot (from the GitLab project):

Wouldn't you rather use a different format for that?

I want to be able to query databases from a VSCode notebook. Given that I want to use notebooks, I don't see a problem with using Markdown notebooks, nor am I aware of a different notebook format that is clearly better.

I don't really understand why markdown files would have outputs

Scenario: Alice is a developer or someone else who is happy to use VSCode notebooks. Bob is a database engineer, with access to a database. Alice wants a report from the database, but is unfamiliar with SQL and doesn't have access to the database. So Bob creates a notebook, writes and executes a query to generate the report, and sends the notebook to Alice. If the result of the query (the output cell) is stored in the notebook, Alice can view the report without having access to the database.
In general, it would be valuable to be able to retain query results on-disk in the notebook file.

Markdown is markdown, I don't expect to open a markdown file and execute the code in it, or store output data, or notebook-specific metadata in my markdown file. I'd recommend using your own format that is specific to this scenario. It's pretty easy to come up with some arbitrary JSON format which would be owned by a notebook serializer in your extension.

I don't expect to open a markdown file and execute the code in it

I thought the point of a notebook, including a markdown notebook, was to execute code. Do you intend some other purpose for markdown notebooks?

I'd recommend using your own format that is specific to this scenario. It's pretty easy to come up with some arbitrary JSON format which would be owned by a notebook serializer in your extension.

I want my notebooks to be human-readable in their raw form. I do not consider JSON to be particularly human readable. Unless I come across some more compelling format, if I were to write my own notebook format it would be something markdown-based.

The main point of this extension is to browse and edit .md files as notebooks. This is using a notebook as a display format for a .md file which is a different goal than using a .md file as a storage format for a real notebook. I'm not sure I want to change the meaning of what a markdown file is and pack extra info into it.
I agree that using .md as a storage format for a notebook is possibly quite nice. JSON is a lot easier to parse :). But if you want to read and write the raw file often, markdown makes sense. Right now I'm thinking that it would make sense to have a different extension and maybe a different file extension for that use case though.


Have "helm uninstall" optionally also delete persistent volume claims

I am quite new to helm and k8s but I found it very confusing. For example I would expect this to work
helm install my-release bitnami/wordpress
helm uninstall my-release
helm install my-release bitnami/wordpress

I would expect this to work but since the bitnami/wordpress chart has persistent volumes it does not. With the second helm install the pods go to a state of crashloopbackoff. What solves it is to manually delete the the persistent volumes after the helm uninstall. It would be good to have an option to also delete those with helm uninstall

Duplicate of #5156.


[VM] Listing availability sets requires a resource group

I noticed that the az vm availability-set list command requires a resource group, but I thought that all list commands supported doing subscription-wide queries. Could this param requirement be lifted, and treated as a way to filter the data if desired?

Not all services expose list-all, though some do
@sauryadas, who can comment for how to proceed on this?

@yugangw-msft looks like list-all api is not available for availability-set?

@sauryadas, please ask CRP and update the issue with rough timeline.

Cleaning up old issues from 2016 - 2018. Please reactivate this if it is still a problem.


python311Packages.agate-excel: 0.2.5 -> 0.3.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python311Packages.agate-excel is: Adds read support for excel files to agate
meta.homepage for python311Packages.agate-excel is: https://github.com/wireservice/agate-excel
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.3.0 with grep in /nix/store/jhn15mc0v7wbpcwyad9sv6s6z2mf0zi5-python3.11-agate-excel-0.3.0
found 0.3.0 in filename of file in /nix/store/jhn15mc0v7wbpcwyad9sv6s6z2mf0zi5-python3.11-agate-excel-0.3.0

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath
csvkit
python310Packages.agate-excel
python311Packages.agate-excel

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/jhn15mc0v7wbpcwyad9sv6s6z2mf0zi5-python3.11-agate-excel-0.3.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python311Packages.agate-excel https://github.com/r-ryantm/nixpkgs/archive/ca6ae1f6f45c3d4ced79e17fcef0307c3249e85f.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/ca6ae1f6f45c3d4ced79e17fcef0307c3249e85f#python311Packages.agate-excel

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/jhn15mc0v7wbpcwyad9sv6s6z2mf0zi5-python3.11-agate-excel-0.3.0
ls -la /nix/store/jhn15mc0v7wbpcwyad9sv6s6z2mf0zi5-python3.11-agate-excel-0.3.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

csvkit
python310Packages.agate-excel
python311Packages.agate-excel

Maintainer pings
cc @vrthra for testing.

Result of nixpkgs-review pr 264770 run on x86_64-linux 1

6 packages built:

csvkit
csvkit.dist
python310Packages.agate-excel
python310Packages.agate-excel.dist
python311Packages.agate-excel
python311Packages.agate-excel.dist


Auto register resource providers

This would also need to add more generated code, it would look  like this...
func (client TopicsClient) CreateOrUpdateSender(req *http.Request) (*http.Response, error) {
	return autorest.SendWithSender(client,
		req,
		azure.RegisterResourceProvider(),
		autorest.DoRetryForStatusCodes(client.RetryAttempts, client.RetryDuration, autorest.StatusCodesForRetry...),
		azure.DoPollForAsynchronous(client.PollingDelay),
	)
}

PTAL @jhendrixMSFT @marstr

Before I dive into the code too much here, I want to understand how it relates to this method:
azure-sdk-for-go/arm/resources/resources/providers.go#L252 @ 7362b8b
Is it a case of circular references getting in our way that we wouldn't reuse this?

I would prefer not to reference the SDK in the runtime. It would also be a circular reference.

@marstr brings up a good point, if we're already generating code that does the work then should we adpot it instead.  Obviously we can't reference the Go SDK due to circular dependencies, but perhaps it makes sense to copy/paste this code instead (i.e. would it set us up better for future servicing of this code).

I copied and pasted code, and left out what made not so much sense in this context (like unmarshaling the response body)

After fixing stuff, generated code would look like this...
func (client TopicsClient) CreateOrUpdateSender(req *http.Request) (*http.Response, error) {
	return autorest.SendWithSender(client,
		req,
		azure.RegisterResourceProvider(),
		azure.DoPollForAsynchronous(client.PollingDelay),
	)
}

This rework with the Azure specific client no longer requires changes on senders in generated code. But it will require changes in the clients file.
// Azure EventGrid Management Client
package eventgrid

import (
	"github.com/Azure/go-autorest/autorest"
+	"github.com/Azure/go-autorest/autorest/azure"
)

const (
	// DefaultBaseURI is the default URI used for the service Eventgrid
	DefaultBaseURI = "https://management.azure.com"
)

// ManagementClient is the base client for Eventgrid.
type ManagementClient struct {
-	autorest.Client
+	azure.Client
	BaseURI        string
	SubscriptionID string
}

// New creates an instance of the ManagementClient client.
func New(subscriptionID string) ManagementClient {
	return NewWithBaseURI(DefaultBaseURI, subscriptionID)
}

// NewWithBaseURI creates an instance of the ManagementClient client.
func NewWithBaseURI(baseURI string, subscriptionID string) ManagementClient {
	return ManagementClient{
-		Client:         autorest.NewClientWithUserAgent(UserAgent()),
+		Client: azure.Client{
+			Client: autorest.NewClientWithUserAgent(UserAgent()),
+		},
		BaseURI:        baseURI,
		SubscriptionID: subscriptionID,
	}
}

PTAL @jhendrixMSFT

With this changes, we can use the clients the same as before (the client only has a different type)
eventgridClient.RetryDuration = time.Second
eventgridClient.RetryAttempts = 3
eventgridClient.PollingDelay = time.Second * 10
eventgridClient.PollingDuration = time.Minute * 5
resp, errChan := eventgridClient.CreateOrUpdate(groupName, name, eventgrid.Topic{
	Location: to.StringPtr("westcentralus"),
}, nil)

This is valid code with and without the change mentioned above to generated code.

Reworked code to get rid of Azure client. Very important change to note is I removed the retries from the Do implementation for autorest.Client. I have also fixed the retriable requests bug.
Generated code would look like this, there are no more breaking changes:
func (client TopicsClient) CreateOrUpdateSender(req *http.Request) (*http.Response, error) {
	return autorest.SendWithSender(client,
		req,
+		azure.RegisterResourceProvider(client.Client),
		azure.DoPollForAsynchronous(client.PollingDelay),
	)
}

PTAL @jhendrixMSFT @marstr


Update Version & Previews

Update Version & Previews for 10 releases
EDIT: #109 added most of the releases needed, only 3 more need to be added now.
EDIT 2: 10 releases to be added now.

I'm almost done, I need 5 more releases done and then I'll create a PR and merge it.
The repo im doing this on is: https://github.com/Kendall-Does-Coding-Websites/mogul-christmas
I'm using the second branch, fix#82 to add all the versions and previews.

#109 added most of the releases needed, only 3 more need to be added now.
REOPENING

KendallDoesCoding removed the good first issue label now

As it is moved the wiki, newcomers, can't help with this issue..

As it is moved the wiki, newcomers, can't help with this issue..

which I find really pointless