[QUESTION] Request for Baseline Code for Experiment Reproduction

Hi, thank you very much for your excellent work and open sourcing the code.
I noticed that in the experimental section of ShadowKV paper, you provided results for several baselines, such as Quest and Loki. However, I wasn't able to find the corresponding code for these baselines in the current repository. I want to reproduce some results of these baseline, would you be able to provide the code used for the baselines mentioned in the paper, or perhaps guide me on how to implement them?
Thank you!

Hello, and thank you for your interest in ShadowKV! Here, we provide the PyTorch implementations for both Quest and Loki.
Below, you’ll find the implementation for Quest:
class QuestCache:
    """Use Max and Min landmarks for retrieval"""
    def __init__(self,
        config :object,
        batch_size :int = 1,
        max_length :int = 32*1024,
        device :str = 'cuda:0',
        dtype = torch.bfloat16,
        sparse_budget: int = 2048,
        chunk_size=16,
        ) -> None:

        self.config = config
        self.batch_size = batch_size
        self.max_length = max_length
        self.device = device
        self.dtype = dtype
        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.head_dim = config.hidden_size // config.num_attention_heads
        self.num_attention_heads = config.num_attention_heads
        self.num_key_value_heads = config.num_key_value_heads

        self.sparse_budget = int(sparse_budget)
        self.chunk_size = chunk_size

        self.k_cache_cpu = torch.zeros(
            config.num_hidden_layers,
            batch_size,
            config.num_key_value_heads,
            self.max_length,
            self.config.hidden_size // self.config.num_attention_heads,
            device=self.device,
            dtype=self.dtype
        )

        self.v_cache_cpu = torch.zeros(
            config.num_hidden_layers,
            batch_size,
            config.num_key_value_heads,
            self.max_length,
            self.config.hidden_size // self.config.num_attention_heads,
            device=self.device,
            dtype=self.dtype
        )

        self.num_layers = config.num_hidden_layers
        self.kv_offset = 0
        self.prefill = 0
        self.gen_offset = 0

        self.k_landmark_max = []
        self.k_landmark_min = []

    def print_stats(self):
        print(f"QuestCache | sparse budget {self.sparse_budget} | chunk size {self.chunk_size} | cached {self.kv_offset}")

    def register_k_landmark(self, k_landmark_max, k_landmark_min):
        self.k_landmark_max.append(k_landmark_max.clone())
        self.k_landmark_min.append(k_landmark_min.clone())

    def prefill_kv_cache(self,
            new_k_cache :torch.Tensor,
            new_v_cache :torch.Tensor,
            layer_idx :int,
            ):

        incoming = new_k_cache.shape[-2] # [bsz, num_kv_heads, incoming, head_dim]
        self.prefill = incoming

        self.v_cache_cpu[layer_idx][:, :, :incoming] = new_v_cache.clone()
        self.k_cache_cpu[layer_idx][:, :, :incoming] = new_k_cache.clone()

        self.chunks = incoming // self.chunk_size - 32 // self.chunk_size
        self.select_sets = self.sparse_budget // self.chunk_size

        self.chunk_end = self.chunks * self.chunk_size

        assert self.select_sets * self.chunk_size == self.sparse_budget, f"({self.select_sets}) * {self.chunk_size} != {self.sparse_budget}"

        key_states_roped_ctx = new_k_cache[:,:,:self.chunks*self.chunk_size].view(self.batch_size, self.num_key_value_heads, self.chunks, self.chunk_size, self.head_dim)

        k_landmark_max = key_states_roped_ctx.min(dim=-2).values
        k_landmark_min = key_states_roped_ctx.max(dim=-2).values

        # register rest_idxed landmarks to k_landmark
        self.register_k_landmark(k_landmark_max, k_landmark_min)

        if layer_idx == self.num_layers - 1:
            assert self.sparse_budget < incoming
            self.kv_offset += incoming

    def collect_kv(self, layer_idx, query_states):
        self.incoming_q_len = query_states.shape[-2] # 1
        min_cache = repeat_kv(self.k_landmark_min[layer_idx], self.num_key_value_groups)
        max_cache = repeat_kv(self.k_landmark_max[layer_idx], self.num_key_value_groups)
        min_value = min_cache * query_states
        max_value = max_cache * query_states

        heuristic = torch.max(min_value, max_value)
        heuristic = heuristic.sum(dim=-1)

        heuristic = heuristic.reshape(1, self.num_key_value_heads, self.num_key_value_groups, -1)
        heuristic = heuristic.sum(dim=-2, keepdim=True)

        topk_chunk = heuristic.topk(k=self.select_sets, dim=-1).indices

        position_ids = (topk_chunk.unsqueeze(-1) * self.chunk_size + torch.arange(self.chunk_size, device=topk_chunk.device).unsqueeze(0).unsqueeze(0).unsqueeze(0)).view(1, self.num_key_value_heads, -1) # [bsz, 8, select_sets * chunk_size]

        key_ = self.k_cache_cpu[layer_idx].gather(dim=-2, index=position_ids.unsqueeze(-1).expand(-1, -1, -1, self.head_dim))
        value_ = self.v_cache_cpu[layer_idx].gather(dim=-2, index=position_ids.unsqueeze(-1).expand(-1, -1, -1, self.head_dim))

        gen_offset = self.gen_offset if layer_idx == self.num_layers - 1 else self.gen_offset + self.incoming_q_len

        ret_k = torch.cat([key_, self.k_cache_cpu[layer_idx][:,:,self.chunk_end:self.prefill+gen_offset]], dim = 2)
        ret_v = torch.cat([value_, self.v_cache_cpu[layer_idx][:,:,self.chunk_end:self.prefill+gen_offset]], dim = 2)

        return ret_k, ret_v

    def update_kv_cache(self,
            new_k_cache :torch.Tensor,
            new_v_cache :torch.Tensor,
            layer_idx :int,
            ):

        incoming = new_k_cache.shape[-2]
        self.k_cache_cpu[layer_idx][:, :, self.kv_offset:self.kv_offset + incoming].copy_(new_k_cache)
        self.v_cache_cpu[layer_idx][:, :, self.kv_offset:self.kv_offset + incoming].copy_(new_v_cache)

        if layer_idx == self.num_layers - 1:
            self.kv_offset += incoming
            self.gen_offset += incoming

    def clear(self):
        self.k_cache_cpu.zero_()
        self.v_cache_cpu.zero_()
        self.k_landmark_max = []
        self.k_landmark_min = []

        self.kv_offset = 0
        self.prefill = 0
        self.gen_offset = 0

    def get_kv_len(self):
        return self.kv_offset

For Loki, you can obtain the projection matrix by either performing offline PCA calibration on a dataset or conducting online decomposition, which is data-dependent and generally more accurate. Below is an example code to illustrate the basic approach. Note that this code is intended for accuracy testing and is not optimized for efficient serving. It’s important to mention that, unlike our approach, Loki applies PCA on a per-head basis.
    def collect_kv(self, layer_idx, query_states, func):
        A = self.A[layer_idx] # torch.Size([8, seq, 16])
        B = self.B[layer_idx] # torch.Size([8, 16, 128])
        self.incoming_q_len = query_states.shape[-2]
        recon_k = torch.einsum('bsd,bdD->bsD', A, B).unsqueeze(0) # low rank recon key  # [1, 8, seq, 128]

        attn_weights = torch.einsum('bhgqd,bhdc->bhgqc', query_states.view(-1, self.num_key_value_heads, self.num_key_value_groups, self.incoming_q_len, self.head_dim), recon_k.transpose(2, 3)) / math.sqrt(128)
        attn_weights = nn.functional.softmax(attn_weights.squeeze(2), dim=-1, dtype=torch.float32).to(self.dtype)

        attn_weights = attn_weights.sum(dim = -2) # [bsz, 8, 4, chunks]
        if self.num_key_value_groups > 1:
            attn_weights, _ = torch.max(attn_weights, dim=-2)
        merged_results = torch.topk(attn_weights, k=self.select_sets, dim=-1).indices

        position_ids = merged_results.unsqueeze(-1).view(1, self.num_key_value_heads, -1)  # [bsz, 8, select_sets * chunk_size]
        value_ = self.v_cache_backup[layer_idx].gather(dim=-2, index=position_ids.unsqueeze(-1).expand(-1, -1, -1, self.head_dim))

        if layer_idx == self.num_layers - 1:
            cur_len = self.kv_offset
        else:
            cur_len = self.kv_offset + self.incoming_q_len

        recent_budget = cur_len - self.prefill
        ret_v = torch.cat([value_, self.v_cache_backup[layer_idx][:,:,self.prefill:cur_len]], dim = 2)

        # gather ret_k
        self.k_cache_gen[layer_idx][:,:,:self.sparse_budget].copy_(self.k_cache_backup[layer_idx].gather(dim=-2, index=position_ids.unsqueeze(-1).expand(-1, -1, -1, self.head_dim)))
        ret_k = self.k_cache_gen[layer_idx][:, :, :self.sparse_budget + recent_budget].clone()

        return ret_v, ret_k

Thank you, I have successfully reproduced this code on the GPU! For the quest baseline, how can I offload all the key-value (kv) pairs to the CPU, and how can I offload only the value? Could you provide the relevant code, as well as the ”layer_compute“ function？

The code provided above is for accuracy evaluation, allowing you to directly adjust Quest’s sparse budget to simulate the budget fetched from the CPU during decoding. For instance, if the sparse budget for ShadowKV is set to 2K, then setting it to 1K with both KV offloaded or 2K with only the value cache offloaded in Quest will result in similar data fetching overhead, though the latter will consume significantly more GPU memory.
You can find the layer_compute function here:
https://github.com/bytedance/ShadowKV/blob/a21f5a9ebd7956fe3510b0f29cf13c4fe07ca3b1/models/base.py#L102

OK,thanks!

The code provided above is for accuracy evaluation, allowing you to directly adjust Quest’s sparse budget to simulate the budget fetched from the CPU during decoding. For instance, if the sparse budget for ShadowKV is set to 2K, then setting it to 1K with both KV offloaded or 2K with only the value cache offloaded in Quest will result in similar data fetching overhead, though the latter will consume significantly more GPU memory.
You can find the layer_compute function here:
https://github.com/bytedance/ShadowKV/blob/a21f5a9ebd7956fe3510b0f29cf13c4fe07ca3b1/models/base.py#L102

OK,thanks!

Hi, I have another question: In the paper "Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference", it is mentioned that Quest skips the first two layers in the KV selection algorithm due to their low sparsity ratio. Could you point out where this is reflected in your code?

You don’t necessarily need to skip the first two layers for some models—it depends on whether their attention distribution is uniform. For instance, Llama-2 has a relatively uniform distribution in the first two layers, so Quest skips them.
However, if you can achieve good accuracy without skipping these layers, that’s preferable (especially when context grows). Some approaches, like TriForce, also avoid skipping the first two layers in the implementation for efficiency.

Based on the experimental results you provided, ShadowKV outperforms Quest on both the ruler and longbench benchmarks, which is quite impressive!  As I understand, longbench includes a wider variety of tasks, so I also  would like to evaluate the performance of ShadowKV on these tasks as well. However, I was unable to find any files related to longbench in the Git repository. Could you kindly provide a brief procedure to reproduce the experiments?

This is a minimal code implementation for LongBench. Interestingly, we observed that LongBench’s performance is influenced by the prompt templates. In some cases, including instructions leads to better performance, while in others, excluding them proves more effective. Therefore, for each method, we report the maximum performance observed across both scenarios.
    elif 'long_bench' in self.dataset_name:
            task = self.dataset_name.split('/')[-1]
            dataset = load_dataset('THUDM/LongBench', task, split='test')

            if self.num_samples > 0:
                self.num_samples = min(self.num_samples, len(dataset))
            else:
                self.num_samples = len(dataset)
            tokenized_prompts = []
            gt = []
            classes = []

            for i in range(len(dataset)):
                if 'llama-3' in self.tokenizer.name_or_path.lower():
                    # model_template = Templates['llama-3'].format(ctx=LONG_BENCH_TEMPLATE[task])
                    model_template = LONG_BENCH_TEMPLATE[task]
                elif 'yi' in self.tokenizer.name_or_path.lower():
                    # model_template = Templates['lwm'].format(ctx=LONG_BENCH_TEMPLATE[task])
                    model_template = LONG_BENCH_TEMPLATE[task]
                elif 'glm' in self.tokenizer.name_or_path.lower():
                    # model_template = Templates['glm'].format(ctx=LONG_BENCH_TEMPLATE[task])
                    model_template = LONG_BENCH_TEMPLATE[task]
                else:
                    raise Exception("Model not found", self.tokenizer.name_or_path)

                input_text = model_template.format(**dataset[i])
                # input_ids = truncate_by_tokens(input_text, self.tokenizer, self.datalen)
                input_ids = self.tokenizer.encode(input_text, return_tensors="pt")

                if input_ids.shape[-1] <= self.datalen and input_ids.shape[-1] > 4096:
                    tokenized_prompts.append(input_ids)
                    gt.append(dataset[i]['answers'])
                    classes.append(dataset[i]['all_classes'])

LONG_BENCH_TEMPLATE = {
    "narrativeqa": "You are given a story, which can be either a novel or a movie script, and a question. Answer the question asconcisely as you can, using a single phrase if possible. Do not provide any explanation.\n\nStory: {context}\n\nNow, answer the question based on the story asconcisely as you can, using a single phrase if possible. Do not provide any explanation.\n\nQuestion: {input}\n\nAnswer:",
    "qasper": "You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write \"unanswerable\". If the question is a yes/no question, answer \"yes\", \"no\", or \"unanswerable\". Do not provide any explanation.\n\nArticle: {context}\n\n Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write \"unanswerable\". If the question is a yes/no question, answer \"yes\", \"no\", or \"unanswerable\". Do not provide any explanation.\n\nQuestion: {input}\n\nAnswer:",
    "multifieldqa_en": "Read the following text and answer briefly.\n\n{context}\n\nNow, answer the following question based on the above text, only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:",
    "multifieldqa_zh": "阅读以下文字并用中文简短回答：\n\n{context}\n\n现在请基于上面的文章回答下面的问题，只告诉我答案，不要输出任何其他字词。\n请用中文回答。\n问题：{input}\n回答：",
    "hotpotqa": "Answer the question based on the given passages. Only give me the answer and do not output any other words.\n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. Only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:",
    "2wikimqa": "Answer the question based on the given passages. Only give me the answer and do not output any other words.\n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. Only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:",
    "musique": "Answer the question based on the given passages. Only give me the answer and do not output any other words.\n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. Only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:",
    "dureader": "请基于给定的文章回答下述问题。\n\n文章：{context}\n\n请基于上述文章回答下面的问题。\n请用中文回答。\n问题：{input}\n回答：",
    "gov_report": "You are given a report by a government agency. Write a one-page summary of the report.\n\nReport:\n{context}\n\nNow, write a one-page summary of the report.\n\nSummary:",
    "qmsum": "You are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences.\n\nTranscript:\n{context}\n\nNow, answer the query based on the above meeting transcript in one or more sentences.\n\nQuery: {input}\nAnswer:",
    "multi_news": "You are given several news passages. Write a one-page summary of all news. \n\nNews:\n{context}\n\nNow, write a one-page summary of all the news.\n\nSummary:",
    "vcsum": "下面有一段会议记录，请你阅读后，写一段总结，总结会议的内容。\n请用中文回答。\n会议记录：\n{context}\n\n会议总结：",
    "trec": "Please determine the type of the question below. Here are some examples of questions.\n\n{context}\n{input}",
    "triviaqa": "Answer the question based on the given passage. Only give me the answer and do not output any other words. The following are some examples.\n\n{context}\n\n{input}",
    "samsum": "Summarize the dialogue into a few short sentences. The following are some examples.\n\n{context}\n\n{input}",
    "lsht": "请判断给定新闻的类别，下面是一些例子。\n请用中文回答。\n{context}\n{input}",
    "passage_count": "There are some paragraphs below sourced from Wikipedia. Some of them may be duplicates. Please carefully read these paragraphs and determine how many unique paragraphs there are after removing duplicates. In other words, how many non-repeating paragraphs are there in total?\n\n{context}\n\nPlease enter the final count of unique paragraphs after removing duplicates. The output format should only contain the number, such as 1, 2, 3, and so on.\n\nThe final answer is: ",
    "passage_retrieval_en": "Here are 30 paragraphs from Wikipedia, along with an abstract. Please determine which paragraph the abstract is from.\n\n{context}\n\nThe following is an abstract.\n\n{input}\n\nPlease enter the number of the paragraph that the abstract is from. The answer format must be like \"Paragraph 1\", \"Paragraph 2\", etc.\n\nThe answer is: ",
    "passage_retrieval_zh": "以下是若干段落文字，以及其中一个段落的摘要。请确定给定的摘要出自哪一段。\n\n{context}\n\n下面是一个摘要\n\n{input}\n\n请输入摘要所属段落的编号。请用中文回答。答案格式必须是\"段落1\"，\"段落2\"等格式\n\n答案是：",
    "lcc": "Please complete the code given below. \n{context}Next line of code:\n",
    "repobench-p": "Please complete the code given below. \n{context}{input}Next line of code:\n"
}

Cool! Thank you very much for providing the code! Additionally, I would greatly appreciate it if you could share more details and code regarding the implementation of InfiniteBench. I would be extremely grateful! By the way, the ShadowKV repository provides an excellent framework for efficient inference with extremely long context inputs (>128K tokens or more). Do you have plans to include the complete code for additional benchmarks (such as LongBench or InfiniteBench) in the repository in the future?

Here is the related code for InfiniteBench. While we may consider adding more benchmarks in the future, this will depend on our bandwidth. Additionally, we believe that RULER is currently the most comprehensive benchmark for long-context LLMs and should serve as the standard.
        elif 'infini_bench' in self.dataset_name: # infini_bench/xxx
            task = self.dataset_name.split('/')[-1]
            dataset = load_dataset("json", data_files=f'{DATADIR["infini_bench"]}/{task}.jsonl', split='train', num_proc=16)
            ### {'id':xxx, 'context': xxx, 'input': xxx, 'answer':xxx}
            if self.num_samples > 0:
                self.num_samples = min(self.num_samples, len(dataset))
            else:
                self.num_samples = len(dataset)
            tokenized_prompts = []
            gt = []

            for i in range(len(dataset)):
                if 'llama-3' in self.tokenizer.name_or_path.lower():
                    model_template = VANILLA_INFINI_BENCH_TEMPLATE[task]
                elif 'yi' in self.tokenizer.name_or_path.lower():
                    model_template = VANILLA_INFINI_BENCH_TEMPLATE[task]
                elif 'glm' in self.tokenizer.name_or_path.lower():
                    model_template = VANILLA_INFINI_BENCH_TEMPLATE[task]
                else:
                    raise Exception("Model not found", self.tokenizer.name_or_path)

                input_text = infini_bench_create_prompt(dataset[i], task, model_template)
                input_ids = truncate_by_tokens(input_text, self.tokenizer, self.datalen)

                if input_ids.shape[-1] <= self.datalen:
                    tokenized_prompts.append(input_ids)
                    gt.append(infini_bench_get_answer(dataset[i], task))

                if len(tokenized_prompts) == self.num_samples:
                    break

            return tokenized_prompts, gt

VANILLA_INFINI_BENCH_TEMPLATE = {
    "passkey": "There is an important info hidden inside a lot of irrelevant text. Find it and memorize it. I will quiz you about the important information.\n\n{context}\n\n{input}\n\nThe pass key is",

    "number_string": "There is an important info hidden inside a lot of irrelevant text. Find it. I will quiz you about the important information there.\n\n{context}\n\n{input}\n\nThe sequence of digits is",

    "kv_retrieval": "Extract the value corresponding to the specified key in the JSON object below. A specified key value pair is hidden within the following text. Make sure to memorize it. I will quiz you about the key value pair afterwards.\n\n{context}\n\nWhat is the specified value for '{input}' mentioned in the provided JSON? Please do not reply with the key, but with the value corresponding to the key.The value associated with '{input}' is:",

    "longbook_sum_eng": "Summarize the book below. \n\n{context}\n\nSummary:",

    "longbook_choice_eng": "Read the book and answer the question.\n\n{context}\n\nQuestion: {question}\nA. {OPTION_A}\nB. {OPTION_B}\nC. {OPTION_C}\nD. {OPTION_D}\n\nThe letter of the correct answer is",

    "longbook_qa_eng": "Read the book and answer the question. Be very concise in your answer.\n\n{context}\n\nQuestion: {question}\nAnswer:",

    "longbook_qa_chn": "阅读以下书籍然后回答问题。\n\n{context}\n请用中文回答。\n问题：{question}\n答案：",

    "math_find": "{prefix}\n\n{context}\n\n{input}",

    "code_run": "There is a function called {func} in the following Python code.\n\n{context}\n\nPlease compute the exact value of {func_call}. The value of {func_call} is",

    "code_debug": "Following is a Python code where exactly one of the functions/methods has a deliberate error that makes it crash.\n\n{context}\n\nOptions:\nA. {OPTION_A}\nB. {OPTION_B}\nC. {OPTION_C}\nD. {OPTION_D}\n\nThe correct option is:",

    "longdialogue_qa_eng": "Below is a dialogue script where one random occurrence of a character name is replaced with \"$$MASK$$\", and you should try to guess who that character is.\n\n{context}\n\n{input} Just give the name without other words. Do not give me random numbers or something else. The name that has been replaced with \"$$MASK$$\" is ",
}

def infini_bench_create_prompt(eg: dict, data_name: str, template: str) -> str:
    # Code tasks
    if data_name == "code_run":
        find_result = re.findall(r"func_[0-9]+\(\-?[0-9]+\)", eg['input'])
        func_call = find_result[0]
        func = func_call.split("(")[0]
        return template.format(
            func=func,
            func_call=func_call,
            context=eg["context"],
        )
    elif data_name in ["code_debug", "code_debug_qa"]:
        code = eg["context"]
        if data_name == "code_debug":
            return template.format(
                context=code,
                OPTION_A=eg["options"][0],
                OPTION_B=eg["options"][1],
                OPTION_C=eg["options"][2],
                OPTION_D=eg["options"][3],
            )
        return template.format(
            context=code,
        )
    # Long book tasks
    elif data_name in [
        "longbook_choice_eng",
        "longbook_qa_eng",
        "longbook_sum_eng",
        "longbook_qa_chn",
    ]:
        book = eg["context"]
        if data_name == "longbook_choice_eng":
            return template.format(
                question=eg["input"],
                context=book,
                OPTION_A=eg["options"][0],
                OPTION_B=eg["options"][1],
                OPTION_C=eg["options"][2],
                OPTION_D=eg["options"][3],
            )
        elif data_name == "longbook_qa_eng":
            return template.format(
                question=eg["input"],
                context=book,
            )
        elif data_name == "longbook_sum_eng":
            return template.format(
                context=book,
            )
        elif data_name == "longbook_qa_chn":
            return template.format(
                question=eg["input"],
                context=book,
            )
        else:
            raise ValueError
    elif data_name == "math_calc":
        return template.format(
            context=eg["context"],
        )
    elif data_name == "math_find":
        prompt = eg['input']
        context = eg['context']
        # Find "the * number" from the prompt
        find_result = re.findall(r"The .+ of", prompt)
        assert find_result, f"Cannot find the target number in {prompt}"
        target_number = find_result[0].lower()[:-3]
        # Replace the number with the answer
        prefix = f"What is {target_number} in the following list?"
        return template.format(
            prefix=prefix,
            context=context,
            input=prompt,
        )

    if "content" in eg:
        content = eg["content"]
        del eg["content"]
        eg["context"] = content

    format_dict = {
        "context": eg["context"],
        "input": eg["input"],
    }

    if data_name == "kv_retrieval":
        format_dict["input"] = eg["input"].split('"')[1]

    prompt = template.format(**format_dict)
    return prompt

def infini_bench_get_answer(eg: dict, data_name: str):
    if data_name in ["code_debug", "longbook_choice_eng"]:
        OPTIONS = "ABCD"
        if isinstance(eg["answer"], str):
            ret = [eg["answer"], OPTIONS[eg['options'].index(eg["answer"])]]
        elif isinstance(eg["answer"], list):
            if len(eg["answer"]) == 1:
                ret = [eg["answer"][0], OPTIONS[eg['options'].index(eg["answer"][0])]]
            elif len(eg["answer"]) == 2 and eg["answer"][1] in ['A', 'B', 'C', 'D']:
                ret = eg['answer']
            else:
                raise ValueError
        else:
            raise ValueError
        return ret

    return eg["answer"]

def truncate_input(input: torch.LongTensor, max_length: int, manner="middle"):
    if max_length < 0:
        return input
    if input.shape[-1] <= max_length:
        return input
    if manner == "middle":
        split = max_length // 2
        return torch.cat([input[:, 0:split],input[:, -split:]], dim=-1)
    else:
        return None

def truncate_by_tokens(input, tok, max_tokens, manner: str = "middle"):
    tokens = tok.encode(input, return_tensors="pt")
    len_before = tokens.shape[-1]
    # print(f"# tokens before: {len_before} (max_tokens: {max_tokens})")
    tokens = truncate_input(tokens, max_length=max_tokens, manner=manner)
    len_after = tokens.shape[-1]  # type: ignore
    # print(f"# tokens after: {len_after} (max_tokens: {max_tokens})")
    assert len_after <= len_before
    assert len_after <= max_tokens or max_tokens < 0
    return tokens

Thank you so much for sharing the code for InfiniteBench! I fully agree with your point that RULER is the most comprehensive benchmark for long-context LLMs and should serve as the standard. Based on some of my own experience, I’ve observed that methods performing well on RULER often maintain strong performance across other benchmarks as well.
I greatly appreciate the effort you’ve put into providing these code and sharing your valuable insights and experience. Wishing your team continued success in all your future work!


TypeError: replace() argument 1 must be str, not bytes

nuitka  --version
<IP_ADDRESS>
Commercial: None
Python: 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)]
Flavor: Unknown
Executable: C:\Users\admin\AppData\Local\Programs\Python\Python38\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10
Nuitka-Memory:INFO: Memory usage changed loading module 'main': 256.00 KB (262144 bytes)
Traceback (most recent call last):
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka_main_.py", line 137, in
main()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka_main_.py", line 123, in main
MainControl.main()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka\MainControl.py", line 913, in main
main_module = _createNodeTree(filename=filename)
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka\MainControl.py", line 115, in _createNodeTree
main_module = Building.buildMainModuleTree(
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka\tree\Building.py", line 1118, in buildMainModuleTree
module.setEarlyModules(detectEarlyImports())
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka\freezer\Standalone.py", line 589, in detectEarlyImports
early_modules = tuple(_detectEarlyImports())
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka\freezer\Standalone.py", line 509, in _detectEarlyImports
result = _detectImports(command=import_code, user_provided=False, technical=True)
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\nuitka\freezer\Standalone.py", line 286, in _detectImports
for line in stderr.replace(b"\r", b"").split(b"\n"):
TypeError: replace() argument 1 must be str, not bytes

Seems very much impossible to happen, since the flavor was not detected, can you confirm this is CPython?

I am also suspecting, the locale might influence this. I see no way, that the process execution is not supposed to give bytes, where of course it wouldn't matter, but maybe you have something in environment variables, that affects Python here, such that it will produce str from a pipe, where bytes are normally given, but it looks very impossible, indeed.

Thanks , I tried rebuilding the environment in conda and it's working fine now. There should be some environmental influence, but not sure where the problem is.

I would appreciate if you could provide more information on what Python this originally was.  From the installation path, I would have said it might be a downloaded CPython, can you confirm? And maybe, I assume you are in Asia, correct me if I am wrong, you got something in site or usersite to make life easier or something like that. But possibly also a .pth file in the site-packages could do funny things. All of which, Nuitka ought to protect itself against well already. If it's not a CPython, I would love to know it better.

Closing for lack of feedback.


[TypeScript] FormData can only imported when esModuleInterop flag is set.

I actually don't know exactly what this means, but wouldn't it be possible to implement the index.d.ts file so that FormData can be imported without setting the flag?
Currently, it is exported like this:
export = FormData;
...
declare class FormData extends stream.Readable { ...

Is there any reason why we could not change it to:
export default class FormData extends stream.Readable { ...

Best regards
Noah

What is the reason for the separate export anyways?


VS Professional 2017 v15.7: MSBuild v4.0 is not supported, aborting.

Hello everyone,
Installed<EMAIL_ADDRESS>+ VS 2017 v15.5: MSBuild v4.0 is not supported, aborting.
Updated to<EMAIL_ADDRESS>Working fine.
Updated VS2017 to v15.7: the same error again (MSBuild v4.0 is not supported, aborting).

cordova-windows 5.x had problems with VS2017 in general, that's probably why that is not working.
cordova-windows 6.x should work with VS17, but seems the newer MSBuild somehow trips it up.
Maybe try setting the env vars mentioned here to try to fix this: https://cordova.apache.org/announcements/2018/02/27/cordova-windows-6.0.0.html

Awesome, thanks for the update!
Do you mind sharing what exactly solved the problem? Future readers might appreciate it.

Of course.
I just set the env variable MSBUILDDIR to the path where VS is installed, in my case C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin.


write tests for each function

to both act as unit test and as example in the docs

Now that we (mostly) have autogenerate for types and docs with examples, writing tests should become easier too...?


Upgrade the minimum php version to 8.0 for dispatcher

fix https://github.com/hyperf/hyperf/issues/4238


feat: add initial support for mutation trees

Explanation
Add initial support for mutation trees.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (5157e64) 20.48% compared to head (b414194) 20.48%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #303   +/-   ##
=======================================
  Coverage   20.48%   20.48%
=======================================
  Files          33       33
  Lines        2143     2143
=======================================
  Hits          439      439
  Misses       1678     1678
  Partials       26       26

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


#90 card styling image border

I tried adding image border. Not fully sure if this is correct. Please let me know places I'm missing.

@quynhngandao - Please update this PR to point to branch v1.10 (and update your local branch with the latest updates). Version 1.9 of Microgen has been released.


Improving logic of IdentifiesImports to support interfaces, static property calls, and nullable types

While trying out the unused import detector, I found it didn't pick up usages of quite a few different imports, so when I used the formatter, it removed some lines it shouldn't have.
This PR adds detection for (see tests for cases of each):

Interfaces extending other interfaces (extends)
Fetching a static property (Class::VALUE)
Nullable types on a function (?Class as an argument datatype, instead of Class)

With these changes, I no longer got unused import errors for the above usages, and also verified via new passing unit tests.

nice! Thanks for the PR!


References from the parent directory are broken

Describe the bug
References skip the base directory - instead of https://site/a/thisshouldbehere/reference/main.html#module.main.SomeClass it links https://site/a/reference/main.html#module.main.SomeClass
Expected behavior
../reference/main.html#module.main.SomeClass
To Reproduce
Use the following index.md and reference something from main.md in index.md.
use_directory_urls: false
site_name: "site"
site_dir: "./public"
repo_name: "mrokita/site"

nav:
  - Introduction: index.md
  - Reference:
      - main.py: reference/main.md
markdown_extensions:
  - toc:
      permalink: true
  - toc:
      permalink: true
  - codehilite:
      linenums: true
  - admonition
  - footnotes
  - pymdownx.superfences
  - pymdownx.tasklist
  - pymdownx.emoji
  - pymdownx.tabbed
  - pymdownx.details
theme:
  name: "material"
  language: pl
  palette:
    primary: white
    accent: blue
extra_css:
  - css/mkdocstrings.css
plugins:
  - search
  - mkdocstrings:
      watch:
        - app
      handlers:
        python:
          selection:
            filters:
              - "!^__"
              - "!^__config__"
              - "!^__json_encoder__"
          rendering:
            show_if_no_docstring: true

Dependencies
mkdocstrings == 0.13.1
pytkdocs == 0.7.0

Workaround
Set use_directory_urls to true

Hi again @mRokita, thanks for the report! Do you have your documentation page available somewhere? I would like to see the HTML code.

OK, the function that computes relative URLs was wrong. It was going one level up too much for non-directory URLs.
To go from a/index.html to a/b.html, it's b.html, not ../b.html.
I will release version 0.13.2 soon :slightly_smiling_face: Thanks again @mRokita!

That's great to hear!


Site redesign

We need to make some major changes to the webite. Let's use this issue to track all the ideas that will come.
I'll put down some of them as a start:

[x] use a new jekyll template
[ ]  link to spectrum and slack community
[ ]  add GitHub banner that links to the GitHub's HospitalRun org
[ ] team page
[ ]  extensive wiki with the documentation for non tech people
[ ]  documentation for tech people
[ ]  "used by" section
[ ]  "supported by" section
[ ] "powered by" section
[ ]  better blog experience

use a new jekyll template
We will use this
Why?

Support SCSS
SEO Friendly
Docs section included
Based on 4 Grid System
Super Clear and Clean Layout
Unique interface
Responsive
Free Google Fonts
Free Font Based Icons
W3C Validated


Run CDK update when instance properties change

When a CDK deploy happens, it reads some instance properties as inputs. When the instance properties change, we want to run the CDK automatically to apply those changes.
This should be built into the admin client.

We need to decide whether to always deploy property changes using the CDK, or if we should model whether each property needs the CDK to apply it.
We probably want to delay applying CDK changes until the user has finished specifying properties they want to change.
There's also a problem with the table properties, as the CDK TableStack and DashboardStack load the table properties from a local file. That file needs to be at a path in the local machine that's specified in an instance property. If the instance was deployed from a different machine, or the table properties are no longer at that path for some reason, this will not work.

The current command line interface isn't really ideal for batching up updates to properties. It seems a bit cumbersome to loop around menus typing in property names explicitly.
It'd be good if we could display a list of properties available to update, let you choose one from the list and edit each value. We could still show the full list on the screen for reference as you make changes. Then when you're finished you hit save to apply the changes, using CDK if necessary.
This seems like a separate issue but it might be better to do that first.

There's also a problem with the system test, because that's deployed with a whole separate CDK app. We'd need to either support calling the separate system test CDK app, or split that out as a separate deployment as in https://github.com/gchq/sleeper/issues/331.

It would be nice to be able to track changes to instance and table properties. If we're going to look at how we model the properties, we could think about how we would make that possible.
Right now we have a mutable model where SleeperProperties is a wrapper around java.util.Properties and has setter methods. When those setters are called you also don't know whether those changes will actually be applied successfully.


Tweaks

Review commit by commit, added some questions by commit.

Make Getting Started consistent
Order docs in footer

Rename to match page "About The Docs"
Question 🤔: These seem redundant, consider just "Getting Started"? Add more links?

Edits to Getting Started

Copy edits for consistency
Remove <br>'s for readability
Question  🤔: Should all headers be Title Case? I see Creating an App but Installing A Dependency. I tend to use https://titlecaseconverter.com/, thoughts?

Great job! :dancer:


Ember.Model.Store container attribute triggers Cannot read property '__container__' of undefined on Ember 2.13 production builds

(I'm mostly just submitting this so future people will find something when Googling for this error.)
Production (but not dev) builds of Ember 2.13 seem to error out before anything is rendered with this error:
TypeError: Cannot read property '__container__' of undefined

I don't entirely understand why this is happening, but seems related to this Ember issue:
emberjs/ember.js#15322
It happens when Ember goes to look up the container injections for the route type, sees store:main (registered by Ember Model), and tries to initialize the Ember.Model.Store object for the first time. It then gets into this block and tries to do Ember.getOwner(this).__container__ (I believe this is the store here) where getOwner returns undefined:
var INJECTED_DEPRECATED_CONTAINER_DESC = {
  // stuff...
  get: function () {
    _emberDebug.deprecate('Using the injected `container` is deprecated. Please use the `getOwner` helper instead to access the owner of this object.', false, { id: 'ember-application.injected-container', until: '2.13.0', url: 'http://emberjs.com/deprecations/v2.x#toc_injected-container-access' });
    return this[CONTAINER_OVERRIDE] || _emberUtils.getOwner(this).__container__;
  },
  // more stuff...
}

I hacked around it on our custom fork of Ember Model by just commenting out the container: null line here:
Ember.Model.Store = Ember.Object.extend({
  // Prod builds (but not dev builds) of Ember 2.13 seem to blow up if you
  // define an attribute called `container`.
  // See https://github.com/emberjs/ember.js/issues/15322
  // container: null,

I've verified that fixes the error.
There might be a more correct fix for this that involved making Ember Model work correctly with the new getOwner() container API, or maybe Ember needs to fix the linked issue.

The only correct way forward is to use the Ember.getOwner() API. Right now, your workaround only works because Ember maintains backwards compatibility. If you look at your debug console, one should see deprecation warnings, though.

https://github.com/ebryn/ember-model/releases/tag/v2.18.0


Mpfs ethernet updates

Summary
This PR fixes two issues for ethernet support on MPFS

Fix a possible deadlock
Fix a case when several devices are connected to a single ethernet switch; generate a valid and unique locally administrated MAC address for every device from the SOC serial number.

Impact
Testing
Tested on both aries m100pfsevp and several custom devices

Re-based to pass the CI

Still squashed a fix patch for mpfs_dsn.c, which was accidentally left out from this PR. Sorry for hassle.


Replace crashing KVO with method swizzling

See the issue 390.
Same thing happened in AFNetworking (PR 2411 and 2702).
So I tried to do the same thing in PINRemoteImage using method swizzle instead of KVO.

          1 Warning

:warning:
Any source code changes should have an entry in CHANGELOG.md or have #trivial in their title.

  Generated by :no_entry_sign: Danger

🚫 CI failed with log

Thanks for the information on iOS 11's new session metrics features. I haven't learned about it yet.
I'll try to catch up.

@garrettmoon which version will fix the crash?

So I decided to abandon method swizzling, and update time properties manually. As for session metrics features, I haven't given time to learn more about it. Sorry for that.

          1 Warning

:warning:
Any source code changes should have an entry in CHANGELOG.md or have #trivial in their title.

  Generated by :no_entry_sign: Danger

🚫 CI failed with log

🚫 CI failed with log

About the failure test testResume, I tried to run the test in my local machine. Even use the master branch of original PINRemoteImage I cannot pass the test. That's strange.

🚫 CI failed with log

@xilin that test only passes on slow network and is unreliable, sorry! Ignore that test and I'll make sure to rerun the CI until it passes before we merge.

It seems that KVO is the best option for the feature currently. I'll close the PR and see if there is the other way to do it right.


 I contacted support, the package was listed shortly thereafter, and the next day I got a reply confirming there had been an incident of some sort and it was resolved. Today I uploaded a package and am seeing exactly the same behavior. 2+ hours on and it's still in "validating" status.
The package in question is Flurl.Http.
Is there another incident or is this the new normal?

@tmenier , sorry for the slow response, missed this issue previously. We had several incidents around the validation pipeline in the last month. More details in the blog here: https://blog.nuget.org/20180201/NuGet-package-publishing-workflow-behind-the-scenes.html
I see that Flurl.Http is available, so closing this issue.


[Automation] Generate Fluent Lite from Swagger servicenetworking#

[Automation] Generate Fluent Lite from Swagger servicenetworking#

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure.resourcemanager:azure-resourcemanager-servicenetworking


feat(docs): Note type IDs and compute_note_hash_and_nullifier page

Closes https://github.com/orgs/AztecProtocol/projects/11/views/23?pane=issue&itemId=52988548

Benchmark results
Metrics with a significant change:

note_successful_decrypting_time_in_ms (8): 315 (-27%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit e63f0226 and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
128 txs

l1_rollup_calldata_size_in_bytes
45,956
181,604
724,196

l1_rollup_calldata_gas
227,192
888,272
3,536,024

l1_rollup_execution_gas
320,383
1,006,720
3,756,402

l2_block_processing_time_in_ms
1,186 (-1%)
4,479
17,642 (-1%)

note_successful_decrypting_time_in_ms
:warning: 315 (-27%)
979 (+1%)
3,626

note_trial_decrypting_time_in_ms
29.5 (-29%)
110 (+5%)
168 (-5%)

l2_block_building_time_in_ms
18,228
72,174
290,098

l2_block_rollup_simulation_time_in_ms
13,483
53,374
214,843

l2_block_public_tx_process_time_in_ms
4,718 (+1%)
18,706 (+1%)
74,992 (+1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
15,706 (-2%)
29,986 (-2%)

note_history_successful_decrypting_time_in_ms
2,519 (+10%)
4,679 (+1%)

note_history_trial_decrypting_time_in_ms
109 (+25%)
199 (-20%)

node_database_size_in_bytes
17,281,104
33,194,064

pxe_database_size_in_bytes
29,923
59,478

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
330 (+1%)
44,568
25,313

private-kernel-ordering
197 (+1%)
43,633
16,193

base-rollup
1,454
129,774
933

root-rollup
83.0 (+1%)
4,192
729

private-kernel-inner
434
70,859
25,313

public-kernel-private-input
243 (+1%)
33,703
26,945

public-kernel-non-first-iteration
243 (+2%)
33,745
26,945

merge-rollup
7.89 (+1%)
2,712
933

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
2 leaves
8 leaves
16 leaves
32 leaves
128 leaves
64 leaves
512 leaves
1024 leaves
2048 leaves
8192 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.3 (-1%)
10.9 (-2%)
12.6 (-1%)
16.8 (-3%)
22.9
63.6 (-1%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.9
17.5
23.0
31.6
47.0
143
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.594 (-1%)
0.605 (-2%)
0.537 (-1%)
0.520 (-3%)
0.479
0.440
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
N/A
N/A
N/A
74.4 (-2%)
47.7 (-1%)
239
462 (-1%)
900
3,541

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
N/A
N/A
N/A
159
96.0
543
1,055
2,079
8,223

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
0.460 (-2%)
0.488 (-1%)
0.436
0.432 (-2%)
0.429
0.426

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
N/A
N/A
N/A
104
57.0
351 (-1%)
692 (-1%)
1,372
5,416

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
N/A
N/A
N/A
197
104
691
1,363
2,707
10,771

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
0.491 (-1%)
0.497
0.474 (-1%)
0.474 (-1%)
0.473
0.472

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
57.0
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
94.1
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
0.577
N/A
N/A
N/A
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contracts are deployed in the tx.

Metric
0 deployed contracts
1 deployed contracts

tx_size_in_bytes
16,859
35,536

Transaction processing duration by data writes.

Metric
0 new commitments
1 new commitments

tx_pxe_processing_time_ms
551
1,455 (+1%)

Metric
0 public data writes
1 public data writes

tx_sequencer_processing_time_ms
0.503 (-1%)
583 (+1%)

Docs Preview
Hey there! 👋 You can check your preview at https://65dec11144b7819a541e7b71--aztec-docs-dev.netlify.app


Update for Chrome on Android

Needs an update for Android's Chrome update. The apple-touch-icon has been depreciated.
<link rel="shortcut icon" sizes="196x196" href="nice-highres.png">
<link rel="shortcut icon" sizes="128x128" href="niceicon.png">
Add to Homescreen - Google Chrome Mobile — Google Developers

Seems like Chrome dev docs refer to 192px not 196px:
https://developer.chrome.com/multidevice/android/installtohomescreen
and also in other web app manifest docs:

https://codelabs.developers.google.com/codelabs/add-to-home-screen/#2
https://developers.google.com/web/updates/2014/11/Support-for-installable-web-apps-with-webapp-manifest-in-chrome-38-for-Android
https://developers.google.com/web/fundamentals/engage-and-retain/web-app-manifest/

Where does 196px come from?


FEAT-#629: Added experimental Ray progress bar

Added progress bar features to Modin
Signed-off-by: Noah Kuo<EMAIL_ADDRESS>
What do these changes do?

[ ] commit message follows format outlined here
[ ] passes flake8 modin
[ ] passes black --check modin
[ ] signed commit with git commit -s
[ ] Resolves #?
[ ] tests added and passing

Tqdm needs to be added to requirements.txt so that installs of Modin will work correctly.

We can add it as an optional dependencies, but probably we should wrap the import in a try..except block so that we can throw a useful error message. Modin will work correctly without tqdm if you don't use the progress bar.

It would be interesting to see if the ProgressBar significantly affects execution time, given that python's threading library can be inefficient.

It should not have any effect on the execution time. Threading here is only used for the purposes of displaying the progress bar. It would be good to test, though.


(FACT-824) Reduce calls to 'ip link show' in Facter::Util::IP

Only call get_bonding_interface when it is result is actually used.

CLA signed by all contributors.

:+1:

Can you re-target at the facter/2.x branch?
Backstory: https://groups.google.com/forum/#!msg/puppet-dev/tBqHZB9M308/QqdaldMsJlkJ
Thanks for the contribution!

Done, the new pr is #869


Useless Algorithm

It's impossible to be unbreakable as you claim. Therefore meaning the algorithm does not solve the problem it was made to solve rendering it useless.

You're right Joshua.
It's a mistake to claim scientifically that an encryption algorithm is unbreakable.
But, as I elaborated in detail at front page, any meaningful attack complexity of this algorithm is higher than the atoms in universe.


run_script plan function fails with 'wrong number of arguments (2 for 3)'

➜  bolt --version
0.7.0

➜  cat ./modules/tasks_demo/plans/puppet.pp
plan tasks_demo::puppet(
  String $nodes,
  String $service_action = 'status',
  String $service_name = 'puppet',
) {

  $nodes_array = split($nodes, ',')

  run_script('tasks_demo/install_puppet_5_agent.sh', $nodes_array)
  #run_command('/opt/puppetlabs/bin/puppet --version', $nodes_array)
  #run_task('service', $nodes_array, action => $service_action, name => $service_name)

}

➜  bolt plan run tasks_demo::puppet nodes=<IP_ADDRESS>:32768,<IP_ADDRESS>:32769,<IP_ADDRESS>:32770 --modulepath ./modules -u root -p root
Error: Evaluation Error: Error while evaluating a Function Call, wrong number of arguments (2 for 3) at /Users/ktreese/workspace/puppet/tasks/modules/tasks_demo/plans/puppet.pp:9:3 on node macbook
Exiting because of an error in Puppet code
➜

➜  ls -l ./modules/tasks_demo/
total 0
drwxr-xr-x  4 reesek<PHONE_NUMBER>  136 Nov  9 16:08 files
drwxr-xr-x  6 reesek<PHONE_NUMBER>  204 Nov  9 16:08 plans
drwxr-xr-x  6 reesek<PHONE_NUMBER>  204 Nov  8 11:14 tasks
➜  ls -l ./modules/tasks_demo/files
total 40
-rw-r--r--  1 reesek<PHONE_NUMBER>   2889 Nov  9 16:08 bashcheck
-rw-r--r--  1 reesek<PHONE_NUMBER>  14233 Nov  9 15:40 install_puppet_5_agent.sh

Documentation indicates two positional arguments.

Note: Downgrading to bolt 0.6.1 does not present the same error.

@ktreese yes, this is an oversight that has been fixed in the vendored copy of Puppet - see https://github.com/puppetlabs/puppet/pull/6347 / https://tickets.puppetlabs.com/browse/PUP-8137
I believe this will be shipped shortly in the upcoming 0.8.0 release, so closing the issue.  Please re-open if for some reason your issue is not resolved at that point.
Thanks!


Pre-trained models

Hi,
The model works good and i would like to know if you can share a pre-trained models.
Thanks

@anuragmishracse any updates on this? It would help me as well as training it (even on a GPU) takes a long time.

@jermenkoo Sorry, I don't have access to the weight file now. I'll try to add the weights file, if I'm able to train a new model.


Android OnOff to java

Problem

Onoff level cluster can not be set in java and not notified to java

Change overview

add related code

Testing

chip-tool onoff/level *

hello, does anyone knows what should I do for this PR now? all comment have been fixed. thanks

I think mostly resolving https://github.com/project-chip/connectedhomeip/pull/16525#discussion_r835344018 is what's needed....


Add mapPartitions transformation

This closes issue #12.

Codecov Report

Merging #13 into master will increase coverage by 0.11%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #13      +/-   ##
==========================================
+ Coverage   95.57%   95.68%   +0.11%
==========================================
  Files           5        5
  Lines         113      116       +3
  Branches        2        3       +1
==========================================
+ Hits          108      111       +3
  Misses          5        5

Impacted Files
Coverage Δ

...danielwestheide/kontextfrei/RDDCollectionOps.scala
100% <100%> (ø)
:white_check_mark:

...ielwestheide/kontextfrei/StreamCollectionOps.scala
98.24% <100%> (+0.03%)
:white_check_mark:

...m/danielwestheide/kontextfrei/DCollectionOps.scala
100% <100%> (ø)
:white_check_mark:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f03c196...cc080ee. Read the comment docs.


VSCode liveshare button

We should add a module that integrates with our VSCode desktop extension + liveshare. It would automatically install the VSCode liveshare extension, and add a button that opens a new session. The button would just say "Open Liveshare" and immediately provide a link to share the session.

I don't think there is a straightforward way to install and configure extensions in the VS Code Desktop before launching it. We may need to support this within the coder.remote extension after we make the connection.
@code-asher may have more ideas here.

Yeah we could add it as a dependency of our extension to make it auto-install.

Yeah we could add it as a dependency of our extension to make it auto-install.

Yeah but should be configurable and only if launched through this module.

That does seem tricky, I am not sure we can do optional dependencies.
Maybe we will need a second plugin that we call instead, that has Liveshare and our plugin as a dependency, and then that second plugin calls our first plugin.


Oculus Quest Controllers not worked after last system update

Hello,
The oculus touch controllers do not work in the ALVR app after the last Oculus Quest update.
I had the same problem in my project.
You should add the following line to the AndroidManifest.xml file:

I have changed your ALVR client (APK file).
ALVRClient-v2.4.0-alpha5-modified.zip
Thank you for your Application.
It was very useful to me.
(I'm sorry for my bad english)

Thank you that fixed my issues

For Oculus Quest you can try JackD83's fork.
It works only with Quest, and it has several Quest specific improvements.


Corrupted WaterIF97 medium models basing on WaterIF97_fixedregion

Reported by rfranke on 16 Jan 2013 07:50 UTC
The following test model investigates if a medium model can be instantiated.
model TestMediaWater
  // select Medium
  //package Medium = Modelica.Media.Water.IdealSteam;
  //package Medium = Modelica.Media.Water.ConstantPropertyLiquidWater;
  //package Medium = Modelica.Media.Water.WaterIF97_pT;
  //package Medium = Modelica.Media.Water.WaterIF97_ph;
  //package Medium = Modelica.Media.Water.WaterIF97OnePhase_ph;
  package Medium = Modelica.Media.Water.WaterIF97_R1ph "fail (use of undeclared variable waterConstants)";
  //package Medium = Modelica.Media.Water.WaterIF97_R2ph "fail (use of undeclared variable waterConstants)";
  //package Medium = Modelica.Media.Water.WaterIF97_R3ph "fail (use of undeclared variable waterConstants)";
  //package Medium = Modelica.Media.Water.WaterIF97_R4ph "fail (use of undeclared variable waterConstants)";
  //package Medium = Modelica.Media.Water.WaterIF97_R5ph "fail (use of undeclared variable waterConstants)";
  //package Medium = Modelica.Media.Water.WaterIF97_R1pT "fail (use of undeclared variable waterConstants)";
  //package Medium = Modelica.Media.Water.WaterIF97_R2pT "fail (use of undeclared variable waterConstants)";

  // instantiate selected Medium
  Medium.BaseProperties medium;
equation
  medium.p = Medium.p_default;
  medium.T = Medium.T_default;
  annotation (experiment(stopTime=1));
end TestMediaWater;

This fails for all models basing on WaterIF97_fixedregion. Dymola 2013 FD01 complaints:
Use of undeclared variable waterConstants
In class Modelica.Media.Water.WaterIF97_fixedregion.
File: D:/home/Modelica/svn.Modelica.org/Modelica/Modelica/Media/Water/package.mo, line 1416
Used in component Modelica.Media.Water.WaterIF97_fixedregion.fluidConstants.
File: D:/home/Modelica/svn.Modelica.org/Modelica/Modelica/Media/package.mo, line 5312

The same error is raised if the package Modelica.Media.Water is checked on its own (SVN 8d20967f08715afed1f130fdde4d5471fbac0e7d).
Shouldn't some automatic test on the SVN trunc detect such things as soon as they are introduced?

Migrated-From: https://trac.modelica.org/Modelica/ticket/964

Comment by sjoelund.se on 16 Jan 2013 15:14 UTC
Likely changed in 3071d98cb409abe020ded6c184d9a767dd1422b1, see also #924 for another problem with the changes.
I should also note that I do perform automated tests on the svn trunk every now and then. But only on full experiments, and it seems no experiment used this class.

Comment by rfranke on 17 Jan 2013 10:55 UTC
Most Fluid examples are just using WaterIF97_pT or WaterIF97_ph, resp. their synonyms StandardWaterOnePhase and StandardWater.
The WaterIF97_fixedregion models are useful for optimizing model performance and robustness. We might need more running examples or tests -- at least in the form of the model used to open this ticket.
Take for example WaterIF97_pT. It fails for:
  medium.p = 290e5;
  medium.T = 350+273.15;

(tool independent). Manually defining the region, WaterIF97_R1pT works fine (using e.g. MSL 3.2).
See also the attached plot. The red dot indicates the exemplary bad point (the grey mesh avoids an evaluation at this point).

Comment by anonymous on 17 Jan 2013 11:21 UTC
Rüdiger: do you have any idea how to avoid the failure if the general model is chosen? I would like to understand why this is failing. Does it also fail if the same point is called within a dynamic trajectory? This is a strange behvaior that I have never seen.

Comment by hubertus on 17 Jan 2013 11:27 UTC
The last comment was by me. /Hubertus

Comment by rfranke on 17 Jan 2013 13:13 UTC
Looking into the IF97 standard, 350 °C is at the border of region 1. The general model calls the function
  BaseIF97.Regions.region_pT

and decides that it is already region 3 -- the independent variable changes from pressure to density. Then
  BaseIF97.Inverses.dofpt3

is called and the explicitly programmed iteration to obtain the density fails (Modelica/Media/Water/IF97_Utilities.mo, line 4871):
   assert(error <> 1, "error in inverse function dofpt3: iteration failed")

The inverse function only starts working for temperatures > 352 °C for the used pressure.
There finds a comment in the function dofpt3:
   //this may need improvement!!

:-)

Comment by hubertus on 17 Jan 2013 14:10 UTC
So the error is that the region is determined wrongly: that is in  my point of view even worse than an inverse that does not converge where it is not valid(and non-convergence of the inverse outside of the region of validity of the forward function is no bug in my book). I don't understand why this happens, the region limits are actually explicit in the variables p-T, without inverses of region boundaries or other complicating factors.

Comment by rfranke on 17 Jan 2013 16:12 UTC
Region 1 goes up to including 350 °C and region 3 starts from including 350 °C. This is why it is fine if Media decides it is region 3.
The problem in fact is the inverse function, which only starts functioning for T > 352 °C -- so there is a whole spanning about 2 K in the beginning of region 3.
If the application at hand shall work for temperatures <= 350°C and the solver happens to evaluate a point at 350°C + epsilon, then a reasonable extrapolation of the equations for region 1 is better than switching equations followed by a whole. In such situations the fixedregion models help to increase the robustness.

Comment by hubertus on 6 Mar 2013 22:36 UTC
Fixed all issues reported here, including the dofpt3 iteration.


Links I made for a custom proxy

nvd123s tools LINKS (may go down but add them)
https://nvd123s-utilities.vercel.app
https://apple-trees-apple-trees-85b6bf01.koyeb.app/
I'll post more soon

Also https://ronaldo-farm.vercel.app

Cool links! Unfortunately I do not add vercel.app or koyeb.app links to my list because it's blocked for me and other people

yea and all the Roblox links and literally almost every one of ur links is blocked for me lol.
Can u at least get new Roblox ones or tell us how to make them?
Or just add these links?
Whatever happens, thank you.

Cool links! Unfortunately I do not add vercel.app or koyeb.app links to my list because it's blocked for me and other people

Also how could I make a non vercel or something link for free
is there a way like to get free .org domains or something

I haven't found any new Roblox links, but you can check out other proxy/link lists here: https://github.com/wea-f/ByePassHub/blob/main/mainUnblockers.md#other-lists-like-this-one--games-proxies-movies-and-even-more-links--links-may-or-may-not-work
To make a non-vercel link for free, follow the freedns tutorial in the MakeYourOwnLink file: https://github.com/wea-f/ByePassHub/blob/main/MakeYourOwnLink.md#making-links-via-freedns

Oh freeDNS I gotta try that at home

I haven't found any new Roblox links, but you can check out other proxy/link lists here: https://github.com/wea-f/ByePassHub/blob/main/mainUnblockers.md#other-lists-like-this-one--games-proxies-movies-and-even-more-links--links-may-or-may-not-work
To make a non-vercel link for free, follow the freedns tutorial in the MakeYourOwnLink file: https://github.com/wea-f/ByePassHub/blob/main/MakeYourOwnLink.md#making-links-via-freedns


Correct the updates of the counts in the sidebar

Another overdue patch. Cleans up and fixes missing cases in the updating of subalbum/image/video counts in the sidebar after delete operations. In the process gets rid of a completely unnecessary album.json.num so that in the future we can remove it from the server as well (it's only needed for smart albums).

Rebased on master for CI Checks.


Whitelist modmail threads as mod channels

Currently Modmail threads aren't marked as mod channels. This means that invoking the !user command does now show any existing notes on a user. This means that the moderator then either has to make an explicit infraction search just to make sure, or to move to a mod channel and invoke the command there (which is less desirable as we usually keep all relevant actions inside the Modmail channel).
The suggestion is to therefore whitelist Modmail channels as mod channels. As these channels are dynamic there is no static way to do so. A solution would then be to add a functionality which allows whitelisting an entire category.
This, however, will have the adverse affect of including the incidents channels which are open to Helpers. It would then be necessary to be able to exclude specific channels in the category whitelisting.
In all the proposed solution is to allow whitelisting of entire categories with the exclusion of specific channels.

I don't particularly see the need to exclude #incidents or #incidents-archive, since no one is going to invoke commands in those channels. For the !infractions command group, we just whitelist the entire ModMail category, along with the mod channels:
https://github.com/python-discord/bot/blob/f97a5a62ab2f8e9af8a426b1ac6d9bf4896a75da/bot/cogs/moderation/management.py#L285-L297
For the user command, we currently have this:
https://github.com/python-discord/bot/blob/f97a5a62ab2f8e9af8a426b1ac6d9bf4896a75da/bot/cogs/information.py#L281
We'd have to change it to something like this:
if ctx.channel.id in constants.MODERATION_CHANNELS or ctx.channel.category_id == constants.Categories.modmail:

Or we could just use in_whitelist_check.

yeah, just use in_whitelist_check, and whitelist the whole information extension. There's no reason we shouldn't have it in modmail threads.

Using the check is uglier IMO
        show_verbose = in_whitelist_check(
            ctx,
            channels=constants.MODERATION_CHANNELS,
            categories=[constants.Categories.modmail],
            redirect=None,
            fail_silently=True,
        )

vs.
       show_verbose = (
           ctx.channel.id in constants.MODERATION_CHANNELS
           or getattr(ctx.channel, "category_id", None) == constants.Categories.modmail
       )


feat: implement the Directory type from WDL 1.2.

This commit implements support for the Directory type in WDL 1.2.
Before submitting this PR, please make sure:

[x] You have added a few sentences describing the PR here.
[x] You have added yourself or the appropriate individual as the assignee.
[x] You have added at least one relevant code reviewer to the PR.
[x] Your code builds clean without any errors or warnings.
[x] You have added tests (when appropriate).
[x] You have updated the README or other documentation to account for these
changes (when appropriate).
[x] You have added an entry to the relevant CHANGELOG.md (see
"keep a changelog" for more information).

This is based on #123. When that merges, I'll rebase this PR.

I'll wait to merge this until @adthrasher weighs in.

@adthrasher are you ok with this new input sort order? If I read the code correctly, the ordering is exactly as you defined with Directory getting slotted in right after File.
I don't think this will have much practical impact on us, as I don't intend to start using Directory in the workflows repo (because it prevents result caching). So more of an academic point if anything 🤷

I'm OK with it. In my opinion, it belongs at the top with File, but I don't feel strongly as to which order those should have. So I'm fine with File first, followed by `Directory.


Add Java Programs

Add Some Programs In Java For Java Learners

can you assign me and explain which type of programs i should contribute


adding Lightwell to prototyping tools

Checklist

[x] I made something good today 💐.
[x] I put links in the correct format: [Tool](link) — description.
[x] My description starts with the lower-case letter and ends with the full stop.
[x] I put the tool in the alphabetical order.

Optional

[x] I added the appropriate labels: free, open-source or mac.
[x] I put the tool only to one category.
[x] I followed Contribution Guidelines.

@maxrose hey! Maybe Lightwell will suit better to Design to Code section or No Code Tools?
It's totally fine for me, so we can add your tool the section you will like the most!

@maxrose thanks for adding your tool. If you want to change a section just ping me :)

@LisaDziuba Lightwell would seem to fit well into a few different categories. I was thinking prototyping made the most sense since many of the features are focused around the early prototype running in the app or locally shared on a device.


Problem importing vue-router & vuex from v0.6.0 onward

The Error
Error on command line console:
[vite] Error while resolving node_modules with id<EMAIL_ADDRESS>{ Error: Cannot find module<EMAIL_ADDRESS>
How to create the problem

follow steps as per normal to create vite project using npx
install vite 0.6.0 (from 0.5.1)
install<EMAIL_ADDRESS>add follwing to index.html file

import { createRouter, createWebHistory } from 'vue-router'

npm run dev and problem will appear when you navigate to the page

This is because the dist file of vue-router referenced a source map but didn't include it in the published package.
Closing since it's a vue-router issue.
/cc @posva

I published a new version without the sourcemaps

that was quiclk. thanks!

I just tried again, it is still happening. Not just to vue-router, but also vuex...
Error: Cannot find module<EMAIL_ADDRESS>Error: Cannot find module<EMAIL_ADDRESS>
The path looks wierd

This is causing the exception to be thrown
in the vite distribution, serverPluginModules.js file
console.log('cc')
            const modulePath = resolve_from_1.default(root, id);
console.log('dd', modulePath)

Going up the code (not down)
    app.use(async (ctx, next) => {
        if (!moduleRE.test(ctx.path)) {
            return next();
        }
        const id = ctx.path.replace(moduleRE, '');
console.log('aa', moduleRE, id, ctx.path)

The console error is...
aa /^\/@modules\// vue /@modules/vue
(node:3832) ExperimentalWarning: The fs.promises API is experimental
aa /^\/@modules\// vuex /@modules/vuex
aa /^\/@modules\// <EMAIL_ADDRESS>/<EMAIL_ADDRESS>aa /^\/@modules\// vue-router /@modules/vue-router
[vite] Error while resolving node_modules with id<EMAIL_ADDRESS>{ Error: Cannot find module<EMAIL_ADDRESS>    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at resolveFileName (C:\Users\user\test\vite-eg\node_modules\resolve-from\index.js:29:39)
    at resolveFrom (C:\Users\user\test\vite-eg\node_modules\resolve-from\index.js:43:9)
    at Object.module.exports [as default] (C:\Users\user\test\vite-eg\node_modules\resolve-from\index.js:46:47)
    at app.use (C:\Users\user\test\vite-eg\node_modules\vite\dist\serverPluginModules.js:148:54) code: 'MODULE_NOT_FOUND' }
aa /^\/@modules\// <EMAIL_ADDRESS>/<EMAIL_ADDRESS>[vite] Error while resolving node_modules with id<EMAIL_ADDRESS>{ Error: Cannot find module<EMAIL_ADDRESS>

Are you on Windows?

Yes, I am on windows 10...

getting the same problem and I'm on a mac

Any updates on this issue?


Fix grammar typo in platform/web/js/engine/config.js

There was a grammatical error in the Godot web export. Fixed: platform/web/js/engine/config.js

I merged this together with other typo fixes in 7223c5b54a7958fe8f8176bafe5ea415f0c514a6 and added you as co-author. Thanks for the contribution!


[SharedCache] Fix some memory leaks in shared cache loading

This fixes three sources of leaks:

MMappedFileAccessor::ReadBuffer was returning a heap-allocated DataBuffer, but no callers were ever deleting it. There does not appear to be any reason to heap allocate the DataBuffer as the type is effectively a smart pointer wrapper around BNDataBuffer. It is now returned by value instead.
MMappedFileAccessor::ReadBuffer was allocating a buffer, copying data into it, and then handing that allocation to the DataBuffer constructor. The constructor copies data into a new allocation it owns so this allocation is unnecessary and was being leaked.
SharedCache::Store was appending a JSON array to itself rather than to a similarly named variable. This cyclic data structure was not correctly freed.

Before this change loading a macOS shared cache would leak over 1GB of memory:
leaks Report Version: 4.0, multi-line stacks
Process 34177: 30805047 nodes malloced for 11333150 KB
Process 34177: 2512405 leaks for<PHONE_NUMBER> total leaked bytes.

After this change the leaks are in the 10s of kilobytes and appear to be unrelated to shared cache loading:
leaks Report Version: 4.0, multi-line stacks
Process 69005: 25357627 nodes malloced for 9319529 KB
Process 69005: 240 leaks for 18400 total leaked bytes.

Additionally, the memory footprint of binaryninja after loading the macOS shared cache drops from 8.3GB to 6.9GB.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@0cyn

Looks like these changes were cherry-picked into dev in 0d27f74  and d553396.


Fix broken url in the frontend

[ ] All menue url push user to /about.
[ ] Footer routes don't work.
[x] Links to social don't work.

#4


On cluster, create base of the relative path if not present

I have up to two cluster workflows for a dataset, sadly, for handling 16s and WGS data. I'm unable to split it into two datasets, because this dataset also corresponds to a card on the site, a file with sample details, etc. Because I'm not using ReFlow like I'm supposed to, it's a bit problematic, but I think I mostly have them solved.
The job ReFlow::StepClasses::MirrorToComputeCluster did not support copying to relative paths. This change creates the base relative to cluster workflow dir if not present, letting me have this structure:
./MicrobiomeStudy_${name}/16s_${name}/
./MicrobiomeStudy_${name}/WGS_${name}/

I think it's a sensible change overall but tell me if you think otherwise! Also I don't want to break people's jobs if it's somehow bad so I'd be grateful for a review.
Tested running and undoing with this change, on the Penn cluster.

Turned out not to be a good idea. From slack:
Steve Fischer  7:07 AM
wojtek, a concern i might have, generally, with creating paths that are not present is that it can mask bugs in the workflow which might cause bigger problems.   why is the path not present?   it might be hard to distinguish cases when it is expected to be absent, and those when it is a bug
Wei Li  8:25 AM
I agree with Steve. Usually, a missing path was caused by missing dependencies or silent failures. We should fix those bugs in workflow before proceeding.
Kathryn Crouch  8:43 AM
The usual procedure for this is to create the directory structure on the production server using calls to ReFlow::StepClasses::MakeDataDir and mirror the empty directories to the cluster as you make them. It's a bit cumbersome to code, but fast enough to run. This ensures that all dependencies are met on both the cluster and the production server.  After processing, you only need to mirror the "leaf" directories back because the rest of the directory structure already exists on the production server.
John Brestelli  10:04 AM
There is a related issue here.  There are 2 reasons (i can think of) to mirror the workflow directory on the cluster.  First is that if you want to browse around and find something it matches what you expect from elm.  The other reason is we often (in genomics workflows) had been assuming there was dependent data already mirrored over so we needed to encode the relative path to the dependent data.  A good example of this was mirroring all of nr to cluster one time per workflow run.
My feeling is that moving forward we should avoid encoding dependencies this way.  it makes it impossible to switch clusters mid workflow.  the nr workflow has already  been redesigned and it no longer depends on any existing cluster files. we are moving all DJobTasks to nextflow and my plan is to make them completely independent.
Getting back to Wojtek's initial question.  Thinking about it this way, mirroring the file paths seems unnecessary.  we just need a working directory for nextflow to run (edited)
:heavy_check_mark:
2
Wojtek Bazant  11:26 AM
Thank you all! How I wanted to do it might a bad idea after all, but I'm not sure if I can use the MakeDataDir solution.
I need to have up to two analyses per dataset. I've modified loading from manual delivery so that it creates this structure
./MicrobiomeStudy_${name}/16s_${name}/
./MicrobiomeStudy_${name}/WGS_${name}/
and there are subgraphs for the 16s and WGS parts. It's partially populated, and then each subgraph adds stuff like configs, and does its own CopyToCluster. I don't want either job to copy the whole of ./MicrobiomeStudy_${name} , weird stuff would happen if it did.
There's no reason for this folder to be on the cluster, but I'm trying to make it so, because the job assumes a symmetry between local and cluster directories:
$self->copyToCluster("$workflowDataDir/$relativePath",  $fileOrDir,  "$clusterWorkflowDataDir/$relativePath");
It's quite neat but maybe it would be broadly useful if I changed that?
My current best idea is to make a MicrobiomeWorkflow::StepClasses::CopyToCluster  which does what it needs to do and move on, as there are better things in life, but I'd be grateful for a yet better idea!


Regenerate gir for GString Deref

Corresponds to https://github.com/gtk-rs/gir/pull/1327

@jf2048 What's the status with this one?


Defer to discord for disabling at-everyone in webhooks

Currently, webhooks permit IRC users to send @everyone and @here pings through, as they're not stripped (and webhooks always have this permission).
Discord already has permissions for whether the bot can ping everyone, and the webhook send method has a disableEveryone option, so let's defer to the inbuilt Discord permission for this.
When permissions can't be found (shouldn't happen), defaults to disabling the pings.
Fixes #494. Supersedes #496 and #457.

Coverage increased (+0.2%) to 81.25% when pulling f6d09103239288199c6aed1468848cae08f08f4d on Throne3d:fix/494-webhooks-at-everyone into ef553d37e25e8437f52e3699a4aaa83b48d22708 on reactiflux:master.

Coverage increased (+0.2%) to 81.25% when pulling f6d09103239288199c6aed1468848cae08f08f4d on Throne3d:fix/494-webhooks-at-everyone into ef553d37e25e8437f52e3699a4aaa83b48d22708 on reactiflux:master.

This appears to, when a user with a name partially matching here / everyone, for example "eve" exists in the server, will ping "eve" instead of everyone.

That's standard for Discord pings – if you type @Throne3dtest, and there's a user @Throne3d on your server, it'll ping them with "test" appended (no space). I think it's fine for the behavior to stay like this, because of that.

I'll merge this for now, since it seems quite important for large bridges – we can amend it later if needs be.


Today, (may 27th) was the release of version 3.7 on yaagl but when I click on launch game it still says it doesn't support it

Is your feature request related to a problem? Please describe.
I dont know[...]
Describe the solution you'd like
The solution I'd like is a way to fix this or if I did anything wrong
Describe alternatives you've considered
Checked the game file integrity
Additional context
when I click launch game it says "The launcher currently doesn't support version 3.7.0 currently, please wait for further updates".

https://github.com/3Shain/yet-another-anime-game-launcher/releases/tag/0.1.0-rc.1 you have to manually update

https://github.com/3Shain/yet-another-anime-game-launcher/releases/tag/0.1.0-rc.1 you have to manually update

how do I manually update?


"bad deflate kind" error

Hello! Can't use node-deflate. Trying to use this code:
Deflate.deflate(new Buffer([255, 255, 255, 251]), 'deflate');```

But it fails with error "bad deflate kind".
I were trying to find out what's the matter, so I suppose it's in the line 378 in deflate.cc:
```char* strLevel = *String::AsciiValue(args[idx+0]->ToString());```

I think so, because there is a value in ```args[idx+0]```, but ```strLevel``` is empty (or may be pointer is NULL).
As far as I've got it from the description of String::AsciiValue() (http://bespin.cz/~ondras/html/classv8_1_1String_1_1AsciiValue.html), "conversion to a string fails" and * String::AsciiValue()  returns NULL.
Could you please tell me what the problem could it be?

I never managed to re-produce this problem, so I'm going to close the issue - anyone who has this problem is welcome to re-open it...


fix: toast import and add readable feedback to the user when Google Auth fails

fix the toast import error, improve the structure of the code, and display an appropriate error message when the Google Auth fails.

Summary by CodeRabbit

New Features

Improved error handling during Google sign-in with user-friendly notifications.

Please do the formatting (using prettier)

Inactive PR


Inflater handle empty streams

Hello,
This prevents the issue of Zlib::BufError errors raised when reading
by Inflater empty streams (should fix #624).
Thanks!

@LukaszMaslej seems like a reasonable enough workaround, although there are some RuboCop failures

The test failures appear to be unrelated. Thanks for the fix!


add jitpack() as repository mirror

Expected Behavior
this is intended as a replace for
maven {
  url 'https://jitpack.io'
}

instead of writing this whole code just add it like the mavenCentral() shortcut:
jitpack()

Current Behavior (optional)
you need to provide a custom repository with url:
maven {
  url 'https://jitpack.io'
}

Context
More or less I want to reduce the time needed to add a jitpack repository. I think, that this is relevant enough as many repositories are based on jitpack.

Thank you for your interest in Gradle!
This issue will be closed as a duplicate of

#16310


Custom marker size Android

Hello,
Is there a way to change the size of a custom marker ?
I explain : I have a custom marker, here a simple red box of 64px :

The callout appears to be centered on a 32x32 inside marker. There is the same problem with the anchor.
Here is my code :
render() {
        return (
            <View style={styles.container}>
            </View>
        )
    };
...
container: {
        flex: 1,
        height: 64,
        width: 64,
        backgroundColor: 'red',
    }
})

If I add "position: 'absolute'" to my style, the size is totally ignored and appear to be 32, but the center is good.
I work on an actual device, an Asus Zenfone 2 ZE551ML, 1080x1920px , 400dpi.

As described in #230 PR #258 fixes this


Improve DataFrame#join

This PR will refine whole code of DataFrame#join method (#158).

Refine DataFrame#join effectively using left_outputs/right_outputs options in Arrow::Table#join.

The performance has improved as a result of refactoring !
$ benchmark-driver -o simple --repeat-count 10 benchmark/combine.yml
Iteration per second (i/s):
                                                           HEAD       0.2.3
C01: Inner join on flights_Q1 by carrier                  61.191        0.441
C02: Full join on flights_Q1 by planes                     0.466        0.287
C03: Left join on flights_Q1 by planes                    41.138        0.280
C04: Semi join on flights_Q1 by planes                    58.869       56.924
C05: Anti join on flights_Q1 by planes                   139.121      130.440
C06: Intersection of flights_1_2 and flights_1_3           0.093        0.085
C07: Union of flights_1_2 and flights_1_3                  0.033        0.032
C08: Difference between flights_1_2 and flights_1_3       26.732       26.835
C09: Concatenate flight_Q1 on flight_Q2                   5.000k       1.803k
C10: Merge flights_Q1_right on flights_Q1_left             0.288        0.292


chore: move v2 branch to main

This is wait on https://github.com/GoogleCloudPlatform/cloud-sql-proxy/pull/1327 before we can merge it.

We'll merge this on Monday just to have enough time to release the binaries, correct any configuration issues, etc.


Reuse of Opensearch core and Opensearch Dashboard setup for CI workflow

Signed-off-by: Ryan Liang<EMAIL_ADDRESS>Description
Reuse of Opensearch core and Opensearch Dashboard setup for CI workflow
Category
Enhancement
Issues Resolved

Resolved #1156

Check List

[ ] ~New functionality includes testing~
[ ] ~New functionality has been documented~
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1212 (e0ccd75) into main (b451cb2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1212   +/-   ##
=======================================
  Coverage   71.78%   71.78%
=======================================
  Files          88       88
  Lines        2027     2027
  Branches      269      269
=======================================
  Hits         1455     1455
  Misses        509      509
  Partials       63       63

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

I have been working on this as well to try to lend a hand. I do not think that GitHub Actions will support removing the manual code any further than what @RyanL1997  has already done.
As things are, GitHub actions has two things that prevent us from doing what we want to do. 1) The checkout action always operates from the working directory and always cleans the entire directory when checking out. This means it is impossible to checkout the security-plugin repo during the security action file without a direct implementation of the action in the security-dashboards repo. So when you call the security repo action the checkout action in the workflow will always move back to the working directory (security-dashboards/security-dashboards) and then wipe that directory and download the files from the security repo.
The second issue is even more problematic and cannot be worked around as far as I have found. The checkout action will always execute commands from a checkout repository in the context of the calling repo NOT the called repo (https://docs.github.com/en/actions/using-workflows/reusing-workflows). This means we cannot call gradle build to get the built plugin zip necessary for installing the plugin into opensearch core. Exacerbating this issue, copying the plugin install action into the dashboards repo will not work because of the lack of gradle properties in the dashboards repo.
Ultimately, I believe that Ryan's best option is to just revert back to what he had working.

@peternied @cwperks Here is the update of reusing the security repo's action:
I was trying to solved this error by manually downloading the security plugin and manually creating the directory ./build/distribution[1] to adapt this command in security repo's action. However, this method doesn't work since the checkout step will wipe out the directory, so that we are constantly receiving this error during the checkout step. I also double confirmed with @scrawfor99, and this is the expected behavior for github actions.
Reference:
[1]: https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/3587095716/workflow#L31-L39

@RyanL1997 Would reversing the order of checking out the branch and downloading the zip provide a temporary fix?
I think it would make sense to add a boolean param with default of false to the start-opensearch-with-one-plugin action (maybe called download_plugin_zip) to download the zip of the plugin from ci.opensearch.org. If the param is set to false it would indicate that the plugin zip is provided to the action, usually as the output of ./gradlew assemble of the checked out branch.

@RyanL1997 @scrawfor99 I think the checkout issues was a red herring - I was able to use a slightly modified version of the job and it looks like it nearly mostly worked [1].  The outstanding failures were test issues that I believe are related to this line          echo "plugins.security.unsupported.restapi.allow_securityconfig_modification: true" >> /opensearch/config/opensearch.yml
Checkout what I did here

Security-Dashboards-Plugin updates https://github.com/opensearch-project/security-dashboards-plugin/compare/main...peternied:security-dashboards-plugin:test-action
https://github.com/opensearch-project/security/compare/main...peternied:security:action-tweaks

[1] https://github.com/peternied/security-dashboards-plugin/actions/runs/3594671289/jobs/6053224787

@peternied @RyanL1997 this looks great. I am glad you managed to avoid the checkout problem altogether with this workaround. @peternied do we need to modify the base for the action merged into the security repo or just call yours? Either seems fine but I will make the changes to security if we want to go that route so that Ryan does not have to.

@scrawfor99 I'm testing with my own fork rn for the security repo side. I will let you know or I can do it after I get it to work.

do we need to modify the base for the action merged into the security repo

Please fix the opensearch-project/security action.  I appreciate the confidence in my fork :D

@peternied @cwperks @scrawfor99 Thank you all for the input! I just fixed the workflow by adding the changes (https://github.com/opensearch-project/security/compare/main...RyanL1997:security:security-reusable-action). If we are good with this change, I can create a PR in security repo to apply it. Once it got merged, I can change the reference from my own fork to Security repo and make this PR ready for review.

I just created the fix PR in Security: https://github.com/opensearch-project/security/pull/2290

@RyanL1997 Awesome job here!


submit api service does not log the full tx id

Context

Information
-

Version
commit 86bc08f5491a45376b46d4c068d1740b0919f8ee on master

Platform
Linux

Installation

Steps to Reproduce

Create a valid transaction file and submit it to submit-api

On the client side you will see a 202 response with full transaction id:

cardano-rest/submit-api$ curl -X POST --header "Content-Type:application/cbor" --data-binary @../../cardano-node/genesis-tx.bin http://localhost:8101/api/submit/tx

"1239e0d0186966cbb2ca0ad08d97e0cea38d12799e011b8135dfc69530b767e2"

On the submit api side you will get the shortened id of transaction logged:

cardano-rest/submit-api$ /home/artur/Projects/DB_Sync/cardano-rest/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-submit-api-2.0.0/x/cardano-submit-api/build/cardano-submit-api/cardano-submit-api --config config/local-config.yaml --genesis-file ../../cardano-node/configuration/genesis/genesis.json --socket-path ../../cardano-node/socket/0 --port 8101
[cardano-tx-submit:Info:9] [2020-03-19 14:40:21.29 UTC] Running tx-submit node
[cardano-tx-submit:Info:9] [2020-03-19 14:40:21.29 UTC] NetworkMagic: RequiresMagic 459045235
[cardano-tx-submit:Info:9] [2020-03-19 14:40:21.29 UTC] localInitiatorNetworkApplication: connecting to node via "../../cardano-node/socket/0"
[cardano-tx-submit:Info:10] [2020-03-19 14:40:21.29 UTC] Running tx-submit web server on http://localhost:8101/
[cardano-tx-submit:Info:23] [2020-03-19 14:42:36.18 UTC] txSubmitPost: received 249 bytes
[cardano-tx-submit:Info:23] [2020-03-19 14:42:36.18 UTC] txSubmitPost: 1239e0d0186966cb

Expected behavior
submit api should return the full tx id just like client
Actual behavior
submit api returns the shortened tx id

Resolution

QA

I can't seem to edit this.  Could you add labels byron and priority high, please @ArturWieczorek - then we will pick them up for action.

@ArturWieczorek , reading your issue / comment after stepping back a bit. Is the issue actually that the log output on the cardano-submit-api service is truncated? However, the transaction id returned by the service as a response to the JSON is actually okay?
If that is the case, then this is a non-issue and we can close this ticket right away :blush:

I agree that not logging the full tx id is not a huge issue, but if we did want to full tx id logged, the fix is a trivial one liner.

A truncated log line is totally fine. I'd just like confirmation from Arthur about this before closing the ticket :pray: ?

Its not even truncated, the official Builder formatter prints a shortened version of the hash. If you want the full hash, you just change formatters.

What's the difference between "a shortened version of the hash" and a truncated hash :) ?

The shortened version is whatever the default Builder for TxId is, and that shortens the hash (which from memory removes chars from the middle). There is also a full builder.

Current one is probably Cardano.Crypto.Hashing.mediumHashF and the full on is (I think) hashHexF.

@KtorZ

Is the issue actually that the log output on the cardano-submit-api service is truncated? However, the transaction id returned by the service as a response is actually okay?
If that is the case, then this is a non-issue and we can close this ticket right away

Yes, the transaction id returned by the service as a response to the client is OK and only the log output from the service is shortened.

Closing this issue - let me know if I should open a separate enhancement ticket for printing full hash or this is not needed.

or this is not needed.

:point_up:


fix: flag to disable auto resize

Related: https://github.com/cornerstonejs/cornerstoneTools/issues/1084

:tada: This PR is included in version 4.2.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket: