mysqldump  file import sharding-proxy  insert speed slowly

Sharding-proxy version: 4.0.0-RC3

Problems encountered：It took 24 hours to import 10 million records into sharding-proxy.

Sharding-proxy Config: start.sh -Xmx8g -Xms8g -Xmn4g,server.yaml use default parm

mannul insert sql：insert into pku_influencing_factor VALUES (null,'test','test',123.00)
How did you get an insert of 1 into 4：insert Affected rows: 4, Time: 0.149000s
sql insert into pku_influencing_factor VALUES (0,'test','test',123.00) normal only 1 record

Do you use PreparedStatement and ParameterMarker?

@terrymanu : I didn't used PreparedStatement and ParameterMarker. What are PreparedStatement and ParameterMarker?

@terrymanu Do you know why insert is so slow?

Because if you don't use PreparedStatement, it will parse SQL for every call which are very slow.
ShardingSphere will cache parse result if using PreparedStatement.

@terrymanu There is not much change ,I have used param:serverTimezone=UTC&useSSL=false&characterEncoding=utf8&useServerPrepStmts=true&cachePrepStmts=true

Terry means you can use ‘insert into pku_influencing_factor VALUES (?,?,?,?)’ with PreparedStatement and put the parameters for every record. @lklkxcxc

@kimmking Can you give me a specific example? thank you！

You can google or baidu PreparedStatement

You can google or baidu PreparedStatement to find its usage.

@kimmking PreparedStatement can be use in sql scripts?


Bump dcos-ui package for 1.9.  Adds multi container form.

High level description including:

Adds multiple container create form

Issues
DCOS-
...
Checklist

[ ] Included a test which will fail if code is reverted but test is not
[ ] Followed relevant code rules Rules for Packages and Systemd

If this is a component/package update, include

[x] Change Log from last: link
[x] Test Results: unit
[ ] Code Coverage:

This is what it looks like

cc/ @spahl

@adam-mesos yes, they're flaky because of some issue we are trying to track down with Chrome. I've removed them from here just so it's not confusing. I've also updated the sha as this hasn't merged yet.

@rcorral who else can do a review of this?

Give me a day to update the PR and then I think @orlandohohmeier would be good to review, unless we don't want managers to do so. Then I can find someone else.

@rcorral Could you find somebody/anybody from your team to give a :rocket: ?
It's just a SHA bump in dcos-ui master, which will soon be updated with a newer master SHA. Nothing controversial, as long as we're going forward in time in the same branch (which I verified).

Looks good :shipit:

@adam-mesos can we use #1137 instead?

@rcorral It looks like #1137 doesn't have any reviews, and this train is ready to leave the station. I could drop this PR from the train, but that doesn't seem necessary, since it's still a forward bump of the SHA.
Get your 2 shipits on #1137 and then we can pull it into the next train.


Fix links in comments

This PR removes highlighting of links in comments. But adds an underline to .underlines. Which are also used on links.

Fixes #57

Looks good to me :+1:


wiki/Using-Hooks ambiguity

"created" can't be used as model attribute since there is a model method with same name:
https://github.com/dresende/node-orm/blob/master/lib/orm.js#L193:L195

Thank you, removed that property from the wiki page


Moved image defaults to a10config

This is less flexible than the defaults being in config.py and will likely need to be configured.
Didn't update any tests because the tests still reference the image_defaults attribute.  I was pleased greatly by this.

Fails tox -e pep8, which for some reason doesn't want an end of line at the end of config.py

Replaced by #177 for trivial pepiness


Early-stop and more features

Added early-stop for executions with bad performance.
Added filtered dataset at the output.
Used phi - standard deviation as performance metric

looks good, merged!


[BUG] - Unable to quit the running task such as "search"

Description
A clear and concise description of what the bug is.
When I connect to my ftp client successfully, I try to search a keyword in it using F, but it gets stuck  in the Searching for "xxx" window, and I cannot quit the job or use other command like F10|Q or F1|H, I have to kill termcp process from other temrinal.
Steps to reproduce
Steps to reproduce the bug you encountered

Connect to a client through FTP
navigate to the client directory
F to search something
it gets stuck, unable to do anything else, have to kill it in other terminal

Expected behaviour
A clear and concise description of what you expected to happen.
It should successfully find or not_find the files/directories instead of getting stuck, and it should not prevent other commands
Environment

OS: [e.g. GNU/Linux Debian 10] Arch Linux
Architecture [Arm, x86_64, ...] x86_x64
Rust version: rustc 1.62.0 (a8314ef7d 2022-06-27)
termscp version: 0.9.0
Protocol used : FTP
Remote server version and name: Android Smart Phone, not sure the version and name

Log
Report the snippet of the log file containing the unexpected behaviour.
If there is any information you consider to be confidential, shadow it.
Additional information
Add any other context about the problem here.

It's not really a bug, the problem is that currently, the file transfer actions are blocking since everything runs on the same thread.
I've been thinking about refactoring the UI for a long time and putting the file transfer on another thread as a Service, but this would require a huge refactoring and a lot of time, which I'm not going to invest at the moment.
The only thing you can do is to avoid long-time execution searches.


Indexing can not be completed

I'm testing
01-basic_indexing_and_search.ipynb
on a Windows 10 PC, in Cursor IDE, using Python 3.11.6
Cell:
RAG.index(collection=[full_document], index_name="Miyazaki", max_document_length=180, split_documents=True)
can not be completed after almost an hour!
[Jan 05, 10:46:21] #> Creating directory .ragatouille/colbert\indexes/Miyazaki
#> Starting...

is shown, I restarted kernel after an hour.
The previous cell, which prints the length of full_document, worked properly.

Hi @vanetreg, sorry about that! I believe this is related to the issue making it not work on Google Colab -- everything is currently multiprocessing even when it doesn't need to be, and it hangs in certain environments (outside __main__ in scripts and in some notebook environments like colab).
I don't have a windows machine to try this, but it might be the Windows + Cursor combo. We'll be looking at fixing this shortly (cc @okhat)

I think indexing and search should definitely work on colab?
https://colab.research.google.com/github/stanford-futuredata/ColBERT/blob/main/docs/intro2new.ipynb

I did notice that it works on the main repo, but doesn't with RAGatouille, must be how we handle the Run... I need to track down exactly why, but it actually hangs:
https://colab.research.google.com/drive/1S3s_5FUjzjOCxuwRhdfEdrZoa2LcEsME?usp=sharing

Same issue here, same environment (even Cursor!)

I was using Windows 11, Cursor, Python 10 through WSL... Worked for me. So, may be a windows not in WSL thing.   I gotta say it would be hard for me to imagine not working in WSL on a Windows machine myself at this point.

@bclavie @okhat
pls. note I've never referred to (Google) Colab.
I tested again and while executing first cell:
from ragatouille import RAGPretrainedModel
RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")

I got:
d:\***\ragatouille-venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

so after installing ipywidgets (requirements?!) and restarting Cursor, now without the above written warning,
the index creation cell's run should again be stopped after 10+ mins, having this note:
[Jan 05, 23:16:42] #> Note: Output directory .ragatouille/colbert\indexes/Miyazaki already exists
#> Starting...

After trying to run next cell:
k = 3 # How many documents you want to retrieve, defaults to 10, we set it to 3 here for readability
results = RAG.search(query="What animation studio did Miyazaki found?", k=k)
results

I got error:
**NameError**                                 Traceback (most recent call last)
[d:\Projects\AI_testing\RAGatouille\01-basic_indexing_and_search.ipynb](file:///D:/Projects/AI_testing/RAGatouille/01-basic_indexing_and_search.ipynb) Cell 14 line 2
      [1](vscode-notebook-cell:/d%3A/Projects/AI_testing/RAGatouille/01-basic_indexing_and_search.ipynb#X16sZmlsZQ%3D%3D?line=0) k = 3 # How many documents you want to retrieve, defaults to 10, we set it to 3 here for readability
----> [2](vscode-notebook-cell:/d%3A/Projects/AI_testing/RAGatouille/01-basic_indexing_and_search.ipynb#X16sZmlsZQ%3D%3D?line=1) results = RAG.search(query="What animation studio did Miyazaki found?", k=k)
      [3](vscode-notebook-cell:/d%3A/Projects/AI_testing/RAGatouille/01-basic_indexing_and_search.ipynb#X16sZmlsZQ%3D%3D?line=2) results
NameError: **name 'RAG' is not defined**

Hey, thanks for confirming @MikeRenwick-ICG and shining some light on it being a Windows (non-WSL) issue @jponline77.
@vanetreg While you're not using Google Colab, this is definitely the same multiprocessing issue that's causing it to hang in colab. I believe the issue you're seeing is still the same problem -- RAG isn't defined because the previous cell never actually ran and just timed out. The likely issue is identified (#13), I'll ping you when we get a fix out for it!

from .autonotebook import tqdm as notebook_tqdm```

so after installing ipywidgets (requirements?!)

This is a bit of an annoying warning, but it doesn't negatively impact running anything. To avoid overloading the lib with dependencies one wouldn't use outside a notebook, we don't generally add ipython/notebook related dependencies to requirements, but definitely do install it if you're going to be running notebooks a lot!

@bclavie
don't you use colab and Jupyter notebook expressions interchangeably? :)
Today I tested again this and it really must be Windows related:
after switch on PC, restarting Windows, the first cell, where RAG is defined, dropped the following error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[d:\Projects\AI_testing\RAGatouille\01-basic_indexing_and_search.ipynb](file:///D:/Projects/AI_testing/RAGatouille/01-basic_indexing_and_search.ipynb) Cell 2 line 1
----> [1](vscode-notebook-cell:/d%3A/Projects/AI_testing/RAGatouille/01-basic_indexing_and_search.ipynb#W1sZmlsZQ%3D%3D?line=0) from ragatouille import RAGPretrainedModel
      [3](vscode-notebook-cell:/d%3A/Projects/AI_testing/RAGatouille/01-basic_indexing_and_search.ipynb#W1sZmlsZQ%3D%3D?line=2) RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\ragatouille\__init__.py:2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/__init__.py:2)
      [1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/__init__.py:1) __version__ = "0.0.1c"
----> [2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/__init__.py:2) from .RAGPretrainedModel import RAGPretrainedModel
      [3](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/__init__.py:3) from .RAGTrainer import RAGTrainer
      [5](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/__init__.py:5) __all__ = ["RAGPretrainedModel", "RAGTrainer"]

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\ragatouille\RAGPretrainedModel.py:3](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/RAGPretrainedModel.py:3)
      [1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/RAGPretrainedModel.py:1) from typing import Callable, Optional, Union
      [2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/RAGPretrainedModel.py:2) from pathlib import Path
----> [3](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/RAGPretrainedModel.py:3) from ragatouille.data.corpus_processor import CorpusProcessor
      [4](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/RAGPretrainedModel.py:4) from ragatouille.data.preprocessors import llama_index_sentence_splitter
      [5](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/RAGPretrainedModel.py:5) from ragatouille.models import LateInteractionModel, ColBERT

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\ragatouille\data\__init__.py:1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/__init__.py:1)
----> [1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/__init__.py:1) from .corpus_processor import CorpusProcessor
      [2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/__init__.py:2) from .preprocessors import llama_index_sentence_splitter
      [3](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/__init__.py:3) from .training_data_processor import TrainingDataProcessor

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\ragatouille\data\corpus_processor.py:2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:2)
      [1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:1) from typing import Callable, Optional, Union
----> [2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:2) from ragatouille.data.preprocessors import llama_index_sentence_splitter
      [5](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:5) class CorpusProcessor:
      [6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:6)     def __init__(
      [7](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:7)         self,
      [8](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:8)         document_splitter_fn: Optional[Callable] = llama_index_sentence_splitter,
      [9](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:9)         preprocessing_fn: Optional[Union[Callable, list[Callable]]] = None,
     [10](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/corpus_processor.py:10)     ):

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\ragatouille\data\preprocessors.py:1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/preprocessors.py:1)
----> [1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/preprocessors.py:1) from llama_index import Document
      [2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/preprocessors.py:2) from llama_index.text_splitter import SentenceSplitter
      [5](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/ragatouille/data/preprocessors.py:5) def llama_index_sentence_splitter(documents: list[str], chunk_size=256):

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\__init__.py:13](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/__init__.py:13)
     [10](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/__init__.py:10) from typing import Callable, Optional
     [12](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/__init__.py:12) # import global eval handler
---> [13](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/__init__.py:13) from llama_index.callbacks.global_handlers import set_global_handler
     [14](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/__init__.py:14) from llama_index.data_structs.struct_type import IndexStructType
     [16](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/__init__.py:16) # embeddings

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\callbacks\__init__.py:7](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/__init__.py:7)
      [5](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/__init__.py:5) from .open_inference_callback import OpenInferenceCallbackHandler
      [6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/__init__.py:6) from .schema import CBEvent, CBEventType, EventPayload
----> [7](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/__init__.py:7) from .token_counting import TokenCountingHandler
      [8](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/__init__.py:8) from .utils import trace_method
      [9](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/__init__.py:9) from .wandb_callback import WandbCallbackHandler

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\callbacks\token_counting.py:6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/token_counting.py:6)
      [4](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/token_counting.py:4) from llama_index.callbacks.base_handler import BaseCallbackHandler
      [5](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/token_counting.py:5) from llama_index.callbacks.schema import CBEventType, EventPayload
----> [6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/token_counting.py:6) from llama_index.utilities.token_counting import TokenCounter
      [7](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/token_counting.py:7) from llama_index.utils import get_tokenizer
     [10](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/token_counting.py:10) @dataclass
     [11](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/callbacks/token_counting.py:11) class TokenCountingEvent:

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\utilities\token_counting.py:6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utilities/token_counting.py:6)
      [1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utilities/token_counting.py:1) # Modified from:
      [2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utilities/token_counting.py:2) # https://github.com/nyno-ai/openai-token-counter
      [4](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utilities/token_counting.py:4) from typing import Any, Callable, Dict, List, Optional
----> [6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utilities/token_counting.py:6) from llama_index.llms import ChatMessage, MessageRole
      [7](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utilities/token_counting.py:7) from llama_index.utils import get_tokenizer
     [10](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utilities/token_counting.py:10) class TokenCounter:

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\llms\__init__.py:1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/__init__.py:1)
----> [1](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/__init__.py:1) from llama_index.llms.ai21 import AI[2](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/__init__.py:2)1
      2 from llama_index.llms.anthropic import Anthropic
      [3](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/__init__.py:3) from llama_index.llms.anyscale import Anyscale

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\llms\ai21.py:6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:6)
      [4](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:4) from llama_index.callbacks import CallbackManager
      [5](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:5) from llama_index.llms.ai21_utils import ai21_model_to_context_size
----> [6](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:6) from llama_index.llms.base import llm_chat_callback, llm_completion_callback
      [7](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:7) from llama_index.llms.custom import CustomLLM
      [8](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:8) from llama_index.llms.generic_utils import (
      [9](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:9)     completion_to_chat_decorator,
     [10](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:10)     get_from_param_or_env,
     [11](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/ai21.py:11) )

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\llms\base.py:25](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:25)
     [14](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:14) from llama_index.callbacks import CallbackManager, CBEventType, EventPayload
     [15](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:15) from llama_index.llms.types import (
     [16](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:16)     ChatMessage,
     [17](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:17)     ChatResponse,
   (...)
     [23](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:23)     LLMMetadata,
     [24](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:24) )
---> [25](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:25) from llama_index.schema import BaseComponent
     [28](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:28) def llm_chat_callback() -> Callable:
     [29](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/llms/base.py:29)     def wrap(f: Callable) -> Callable:

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\schema.py:16](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/schema.py:16)
     [13](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/schema.py:13) from typing_extensions import Self
     [15](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/schema.py:15) from llama_index.bridge.pydantic import BaseModel, Field, root_validator
---> [16](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/schema.py:16) from llama_index.utils import SAMPLE_TEXT, truncate_text
     [18](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/schema.py:18) if TYPE_CHECKING:
     [19](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/schema.py:19)     from haystack.schema import Document as HaystackDocument

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\utils.py:89](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:89)
     [85](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:85)             self._stopwords = stopwords.words("english")
     [86](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:86)         return self._stopwords
---> [89](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:89) globals_helper = GlobalsHelper()
     [92](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:92) # Global Tokenizer
     [93](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:93) @runtime_checkable
     [94](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:94) class Tokenizer(Protocol):

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\llama_index\utils.py:45](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:45), in GlobalsHelper.__init__(self)
     [43](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:43) def __init__(self) -> None:
     [44](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:44)     """Initialize NLTK stopwords and punkt."""
---> [45](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:45)     import nltk
     [47](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:47)     self._nltk_data_dir = os.environ.get(
     [48](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:48)         "NLTK_DATA",
     [49](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:49)         os.path.join(
   (...)
     [52](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:52)         ),
     [53](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:53)     )
     [55](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/llama_index/utils.py:55)     if self._nltk_data_dir not in nltk.data.path:

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\nltk\__init__.py:180](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/__init__.py:180)
    [177](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/__init__.py:177) else:
    [178](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/__init__.py:178)     from nltk import cluster
--> [180](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/__init__.py:180) from nltk.downloader import download, download_shell
    [182](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/__init__.py:182) try:
    [183](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/__init__.py:183)     import tkinter

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\nltk\downloader.py:2479](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2479)
   [2469](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2469)             pass
   [2472](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2472) ######################################################################
   [2473](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2473) # Main:
   [2474](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2474) ######################################################################
   (...)
   [2477](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2477)
   [2478](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2478) # Aliases
-> [2479](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2479) _downloader = Downloader()
   [2480](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2480) download = _downloader.download
   [2483](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:2483) def download_shell():

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\nltk\downloader.py:515](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:515), in Downloader.__init__(self, server_index_url, download_dir)
    [513](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:513) # decide where we're going to save things to.
    [514](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:514) if self._download_dir is None:
--> [515](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:515)     self._download_dir = self.default_download_dir()

File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\nltk\downloader.py:1072](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1072), in Downloader.default_download_dir(self)
   [1069](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1069) # Check if we have sufficient permissions to install in a
   [1070](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1070) # variety of system-wide locations.
   [1071](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1071) for nltkdir in nltk.data.path:
-> [1072](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1072)     if os.path.exists(nltkdir) and nltk.internals.is_writable(nltkdir):
   [1073](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1073)         return nltkdir
   [1075](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1075) # On Windows, use %APPDATA%

AttributeError: partially initialized module 'nltk' has no attribute 'internals' (most likely due to a circular import)

Note
[1075](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/nltk/downloader.py:1075) # On Windows, use %APPDATA%
at the end of error message.

I was using Windows 11, Cursor, Python 10 through WSL... Worked for me. So, may be a windows not in WSL thing. I gotta say it would be hard for me to imagine not working in WSL on a Windows machine myself at this point.
@jponline77
I tested it both in VSC and Cursor, in both WSL extension installed.
Maybe Windows version (10 / 11 ) matters?

anyone tried this outside of windows Jupyter?

Hi @bclavie,
I'm not sure if the problem is related to Colab, I also have an error using Jupyter locally on my Ubuntu server.
The basic readme.md example doesn't work and the cell never finish executing.
Here's the code and stacktrace if that helps:
from ragatouille import RAGPretrainedModel

RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
my_documents = [
    "This is a great excerpt from my wealth of documents",
    "Once upon a time, there was a great document"
]

index_path = RAG.index(index_name="my_index", collection=my_documents)

output the following:
[Jan 06, 10:41:35] #> Creating directory .ragatouille/colbert/indexes/my_index

#> Starting...
#> Starting...
nranks = 2 	 num_gpus = 2 	 device=1
[Jan 06, 10:41:38] [1] 		 #> Encoding 0 passages..
nranks = 2 	 num_gpus = 2 	 device=0
[Jan 06, 10:41:38] [0] 		 #> Encoding 2 passages..

 File "/home/np/miniconda3/envs/np-ml/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py", line 101, in setup
    avg_doclen_est = self._sample_embeddings(sampled_pids)
  File "/home/np/miniconda3/envs/np-ml/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py", line 140, in _sample_embeddings
    self.num_sample_embs = torch.tensor([local_sample_embs.size(0)]).cuda()
AttributeError: 'NoneType' object has no attribute 'size'

Hey @timothepearce, thanks for flagging! I believe this is a very separate problem (the multiprocessing in your case runs fine, but there seems to be another problem). Could you create a new issue so I can look into it a bit more? And could you try out the notebooks in examples/ ? I think there might be something wrong with the README, which is (probably) that there aren't enough documents in the example (which I could fix by adopting a separate logic for n_docs that are far too small).

@runonthespot Feel free to try https://github.com/bclavie/RAGatouille/blob/main/examples/01-basic_indexing_and_search.ipynb, it's fully plug-and-play!

@bclavie You're right, the code doesn't work either in the Python CLI, and seems related to the ColBERT library.
I'll open a new issue and dig a little bit more.

Hey @vanetreg, for your other issue, the partial init -- no idea what's going on there, it seems like something weird happened when initialising ntlk?
I've tested some things on my end and I can confirm this is due to how ColBERT does multiprocessing, which causes the issue in some environments (seemingly Colab and Windows 10). This will eventually be fixed once the multiprocessing handling is changed upstream but sadly there doesn't seem to be a good in-notebook workaround on those two platforms at the moment.
If you use RAGatouille in a python script (making sure to have it inside if __name__ == "__main__":), it should hopefully run fine (though again, not tested on Windows)!

Hey @bclavie ,
I'm gonna test it on Replit during weekend and be back with the result.

Hey @bclavie , I'm gonna test it on Replit during weekend and be back with the result.

Hey @bclavie
after 2 days I found out Replit  doesn't handle Jupyter notebooks properly, so I'm not able to test RAGatouille there, so since Google Colab also isn't an option, I should wait for the Windows fix :)

I was using Windows 11, Cursor, Python 10 through WSL... Worked for me. So, may be a windows not in WSL thing. I gotta say it would be hard for me to imagine not working in WSL on a Windows machine myself at this point.

@jponline77 I tested it both in VSC and Cursor, in both WSL extension installed. Maybe Windows version (10 / 11 ) matters?

Yeah, maybe it's a Windows 10 issue.  Just be sure, if you are using WSL, that it's actually running in WSL.  If you are setup to run in WSL, then you should be able to try to run it command line from WSL directly without using VSC or Cursor.  My experience with WSL is that it runs everything that runs in Ubuntu in a very similar way as if it was a standalone Linux system.  So, it would surprise me a little if it matters if you are Windows 10 or 11.  That said, any reason you aren't interested in upgrading to 11?  I've now got RAGatouille running on two different systems with Windows 11 and WSL.  One was a Laptop with a low end integrated GPU and 16GB of memory.  It did take 10 minutes to index a small file but it worked.

@jponline77
I won't upgrade this PC, neither HW or Windows 10, so if I don't find a proper online IDE / runtime ( payed for Replit Core, considering Google Colab Pro ) with optional GPUs, I'll go for an M2 Mac mini :)

Hey, thanks for this @jponline77 -- indexing is slow sadly, taking a while to create the index is the tradeoff to querying very large corpuses at near-constant time. It can maybe be optimised though (that'd require work on the upstream ColBERT repo), but that's something for the future!
I'm working on a feature to do index-free search, it's not very scaleable, at least at the moment (you could query maybe up to 1k documents in >1s on a T4 GPU, and obviously much slower every time you add something) but for smaller corpuses it will make it easy to try it out!

@vanetreg I think (not sure) you could try it out in a standalone script like I mentioned earlier? Wrap it in if __name__ == "__main__":... It's not ideal for interactivity but it could work! (At least it does on every non-windows platform I've tried). Anyhow, the Mac Mini is an excellent choice 😄

Hey, thanks for this @jponline77 -- indexing is slow sadly, taking a while to create the index is the tradeoff to querying very large corpuses at near-constant time.

I was actually a little surprised it worked at all on the laptop.  Indexing speed was much faster on my RTX4080 system with 128GB of ram :)

I'm getting it hanging on wsl2 ubuntu (win 11) as well. In a notebook and as a standalone python script (as well as wrapped in main).
Pretty sure it's something to do with nccl, and likely colbert

Multiprocessing is no longer enforced for indexing when using no GPU or a single GPU thanks to @Anmol6's excellent upstream work on https://github.com/stanford-futuredata/ColBERT/pull/290 & propagated by https://github.com/bclavie/RAGatouille/pull/51.
This is likely to fix the indexing problems on Windows (or at least, one of the problems). Please let me know if the latest version of RAGatouille fixes it for you!

@bclavie
I updated to latest
0.0.4b2
(having Win 10, python 3.11.6)
and after loading dotenv ( setting HF_HUB_DISABLE_SYMLINKS_WARNING=true )
I still have errors while running:
from ragatouille import RAGPretrainedModel
RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")

having error messages:
CalledProcessError                        Traceback (most recent call last)
Cell In[3], [line 3](vscode-notebook-cell:?execution_count=3&line=3)
      [1](vscode-notebook-cell:?execution_count=3&line=1) from ragatouille import RAGPretrainedModel
----> [3](vscode-notebook-cell:?execution_count=3&line=3) RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
...
File [d:\Projects\AI_testing\RAGatouille\ragatouille-venv\Lib\site-packages\torch\utils\cpp_extension.py:2382](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/torch/utils/cpp_extension.py:2382), in _write_ninja_file(path, cflags, post_cflags, cuda_cflags, cuda_post_cflags, cuda_dlink_post_cflags, sources, objects, ldflags, library_target, with_cuda)
   [2380](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/torch/utils/cpp_extension.py:2380) link_rule = ['rule link']
   [2381](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/torch/utils/cpp_extension.py:2381) if IS_WINDOWS:
-> [2382](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/torch/utils/cpp_extension.py:2382)     cl_paths = subprocess.check_output(['where',
   [2383](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/torch/utils/cpp_extension.py:2383)                                         'cl']).decode(*SUBPROCESS_DECODE_ARGS).split('\r\n')
   [2384](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/torch/utils/cpp_extension.py:2384)     if len(cl_paths) >= 1:
   [2385](file:///D:/Projects/AI_testing/RAGatouille/ragatouille-venv/Lib/site-packages/torch/utils/cpp_extension.py:2385)         cl_path = os.path.dirname(cl_paths[0]).replace(':', '$:')

File [~\AppData\Local\Programs\Python\Python311\Lib\subprocess.py:466](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:466), in check_output(timeout, *popenargs, **kwargs)
    [463](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:463)         empty = b''
    [464](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:464)     kwargs['input'] = empty
--> [466](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:466) return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    [467](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:467)            **kwargs).stdout

File [~\AppData\Local\Programs\Python\Python311\Lib\subprocess.py:571](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:571), in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    [569](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:569)     retcode = process.poll()
    [570](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:570)     if check and retcode:
--> [571](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:571)         raise CalledProcessError(retcode, process.args,
    [572](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:572)                                  output=stdout, stderr=stderr)
    [573](https://file+.vscode-resource.vscode-cdn.net/d%3A/Projects/AI_testing/RAGatouille/~/AppData/Local/Programs/Python/Python311/Lib/subprocess.py:573) return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['where', 'cl']' returned non-zero exit status 1.

I think this is an issue with Windows 10 and loading cpp extensions in PyTorch? Saw a few similar issues on other projects floating around... I think the current stance will be that the lib doesn't support Win10 unless someone can figure out a solid fix to this 😞

In case others are trying to get it working on Windows 10, I did get past the cl error with non-zero exit status above (by installing the C++ parts of VS 2022 Build Tools) but I then ran into issues with phtread.h not being found. I tried vcpkg to install it (which was possible) but I still couldn't get it to work with the compiler and when I saw that cpp_extensions now seems archived, that, along with the time/effort taken to get to that point made me give up on Windows directly (for now at least!)
However I didn't have any problems with ragatouille using WSL on Windows (Ubuntu 20.04) via pip install ragatouille within a conda env with Python 3.11.7.

I'm using WSL2 Windows 10 with faiss-gpu installed and faiss-cpu uninstalled. I've been here for 30 minutes...
[Feb 13, 18:24:57] #> Will delete 1 files already at .ragatouille/colbert/indexes/Miyazaki in 20 seconds...
[Feb 13, 18:25:18] [0]           #> Encoding 81 passages..
[Feb 13, 18:25:20] [0]           avg_doclen_est =<PHONE_NUMBER>9628906        len(local_sample) = 81
[Feb 13, 18:25:20] [0]           Creating 1,024 partitions.
[Feb 13, 18:25:20] [0]           Estimated 10,516 embeddings.
[Feb 13, 18:25:20] [0]           #> Saving the indexing plan to .ragatouille/colbert/indexes/Miyazaki/plan.json ..
After 30 minutes, I got the error:
WARNING clustering 9991 points to 1024 centroids: please provide at least 39936 training points
Clustering 9991 points in 128D to 1024 clusters, redo 1 times, 20 iterations
Preprocessing in 0.00 s
Faiss assertion 'err == CUBLAS_STATUS_SUCCESS' failed in void faiss::gpu::runMatrixMult(faiss::gpu::Tensor<float, 2, true>&, bool, faiss::gpu::Tensor<T, 2, true>&, bool, faiss::gpu::Tensor<IndexType, 2, true>&, bool, float, float, cublasHandle_t, cudaStream_t) [with AT = float; BT = float; cublasHandle_t = cublasContext*; cudaStream_t = CUstream_st*] at /project/faiss/faiss/gpu/utils/MatrixMult-inl.cuh:265; details: cublas failed (13): (512, 128) x (1024, 128)' = (512, 1024) gemm params m 1024 n 512 k 128 trA T trB N lda 128 ldb 128 ldc 1024
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
Any ideas?

(Copy/pasting this message in a few related issues)
Hey guys!
Thanks a lot for bearing with me as I juggle everything and trying to diagnose this. It’s complicated to fix with relatively little time to dedicate to it, as it seems like the dependencies causing issues aren’t the same for everyone, with no clear platform pattern as of yet. Overall, the issues center around the usual suspects of faiss and CUDA.
While because of this I can’t fix the issue with PLAID optimised indices just yet, I’m also noticing that most of the bug reports here are about relatively small collections (100s-to-low-1000s). To lower the barrier to entry as much as possible, https://github.com/bclavie/RAGatouille/pull/137 is introducing a second index format, which doesn’t actually build an index, but performs an exact search over all documents (as a stepping stone towards https://github.com/bclavie/RAGatouille/issues/110, which would use an HNSW index to be an in-between compromise between PLAID optimisation and exact search).
This approach doesn’t scale, but offers the best possible search accuracy & is still performed in a few hundred milliseconds at most for small collections. Ideally, it’ll also open up the way to shipping lower-dependency versions (https://github.com/bclavie/RAGatouille/issues/136)
The PR above (https://github.com/bclavie/RAGatouille/pull/137) is still a work in progress, as it needs CRUD support, tests, documentation, better precision routing (fp32/bfloat16) etc… (and potentially searching only subset of document ids).
However, it’s working in a rough state for me locally. If you’d like to give it a try (with the caveat that it might very well break!), please feel free to install the library directly from the feat/full_vectors_indexing branch and adding the following argument to your index() call:
index(…
index_type=“FULL_VECTORS”,
)

Any feedback is appreciated, as always, and thanks again!

(Copy/pasting this message in a few related issues)
Hey guys!
Thanks a lot for bearing with me as I juggle everything and trying to diagnose this. It’s complicated to fix with relatively little time to dedicate to it, as it seems like the dependencies causing issues aren’t the same for everyone, with no clear platform pattern as of yet. Overall, the issues center around the usual suspects of faiss and CUDA.
While because of this I can’t fix the issue with PLAID optimised indices just yet, I’m also noticing that most of the bug reports here are about relatively small collections (100s-to-low-1000s). To lower the barrier to entry as much as possible, https://github.com/bclavie/RAGatouille/pull/137 is introducing a second index format, which doesn’t actually build an index, but performs an exact search over all documents (as a stepping stone towards https://github.com/bclavie/RAGatouille/issues/110, which would use an HNSW index to be an in-between compromise between PLAID optimisation and exact search).
This approach doesn’t scale, but offers the best possible search accuracy & is still performed in a few hundred milliseconds at most for small collections. Ideally, it’ll also open up the way to shipping lower-dependency versions (https://github.com/bclavie/RAGatouille/issues/136)
The PR above (https://github.com/bclavie/RAGatouille/pull/137) is still a work in progress, as it needs CRUD support, tests, documentation, better precision routing (fp32/bfloat16) etc… (and potentially searching only subset of document ids).
However, it’s working in a rough state for me locally. If you’d like to give it a try (with the caveat that it might very well break!), please feel free to install the library directly from the feat/full_vectors_indexing branch and adding the following argument to your index() call:
index(…
index_type=“FULL_VECTORS”,
)

Any feedback is appreciated, as always, and thanks again!

The CUBLAS errors turned out to be faiss incompatible driver issues for most people. This should be fixed by the new experimental default indexing in 0.0.8, which skips using faiss (does K-means in pure pytorch) as long as you're indexing fewer than ~100k documents!


Conflicts with SWRevealViewController.

How to use ios-PDF-Reader with SWRevealViewController. i found that both are not working together...
Please help.

https://stackoverflow.com/a/25232124


change doc link + update install instructions

Fixing the docs!
Thanks to @jenshnielsen, I always forget they're not pointing to master.
I just changed to point to master, cause it's better to have them broken sometimes than not updating them for a year.
(Maybe we should automatize this, i don't remember anymore why I didn't do this in the past, but might be a bit of an overhead?)
Thank you @jenshnielsen :)

One issue to be fixed on readthedocs is that the default branch of docs still points to latest
so if you go to
https://ipycytoscape.readthedocs.io/ you end up at latest

Oh, thanks for the catch!
Will fix :)

Done, and also sent an invite for you to have access to this stuff ;)
Sorry I forgot Ian.


SAMZA-1384: Race condition with async commit affects checkpoint correctness

@prateekm @nickpan47 @xinyuiscool @vjagadish1989


Firefox style fix for PricingPlan component

A very small fix for the PricingPlan styles. I've also removed the PricingText component because we don't handle internationalisation so our test env blows up.

 Browser test upgrade page


Condense UI layout

Hello,
Thanks for this fantastic project! I'm submitting some minor interface changes for potential inclusion. I haven't been able to get the project to build successfully yet (will open a separate issue for that if I can't sort it) so I haven't been able to test whether further changes need to be done unfortunately.
I'll mark this as a draft for now, if you are able to confirm that the changes work successfully in the compiled binary and wish to merge please feel free to do so or let me know if there's other changes to be made.

Sweet job! This design is far better than the old one. Thanks for contributing!
I confirm you that the compiled binary works well and that there are no errors.
Merging.
PS: Feel free to open an issue if you can't get it to build.


Issue with multiple IIS applications in one site using SecRemoteRules

So, we're using SecRemoteRules to deploy additional rules to a site after deployment, and for most of our software, this is working great, but one of our products involves multiple web applications that run as part of one web site (in one application pool), and I'm getting some odd behavior.
It looks like rules configured in any application are scoped to that application (ie: if I add an Include directive in the config for one application, including the root, those rules don't apply to others), so it seems reasonable to add the necessary Include directives to each application. Which is fine.
This doesn't work for SecRemoteRules though. Like Include, it doesn't appear to inherit or otherwise be shared, but if I use it in multiple applications in the same site, I get the error: ModSecurity: SecRemoteRules cannot be used more than once.
I have some possible workarounds in mind (one would be just not using SecRemoteRules with this product, but I'm also looking at seeing if breaking it up into multiple application pools helps), but I'm wondering if this behavior is known/intended and if there's any recommended solution here. I'm using ModSecurity for IIS (STABLE)/2.9.2 on IIS 10 on Windows 10.

Hi @wscalf,
No, this behave is not on propose. The SecRemoteRules stores a uri in a global compartment and this compartment is later used to retrieve the rules:
https://github.com/SpiderLabs/ModSecurity/blob/f66cd4111f08a26d4429cb0071e992deb050928a/apache2/msc_remote_rules.c#L670-L671
Later, the rules are parsed:
https://github.com/SpiderLabs/ModSecurity/blob/f66cd4111f08a26d4429cb0071e992deb050928a/apache2/msc_remote_rules.c#L725-L751
And the remote server structure is finally filled with the amount of loaded rules:
https://github.com/SpiderLabs/ModSecurity/blob/f66cd4111f08a26d4429cb0071e992deb050928a/apache2/msc_remote_rules.c#L725-L751
The rules are loaded from/to the compartment that they belong. The only reason why multiples SecRemoteRules are not allowed, is due to the fact that its initial structure is not an array.

So, I'm not sure I'm understanding this response. I see the code and understand that it's an error to use SecRemoteRules more than once, and it's kind of interesting that the reason for it is as simple as the original data structure. It might be useful to have it expanded to be allowed.
My concern, though, is that I have seemingly conflicting behavior-

On the one hand, using SecRemoteRules in more than one website in the same application pool causes this error (presumably because all the websites in an application pool run in one Windows process) and..
On the other hand, remote rules loaded at a higher website aren't inherited by lower ones the way IIS configuration sections that aren't allowed to be specified multiple times are.

Between the two, there doesn't appear to be a way to run multiple websites in one application pool and also load remote rules that affect them all. So, for the time being, we've broken those websites up into their own application pools, so they each get their own Windows processes, and that seems to have helped.
Peculiarly, we're still getting this error sometimes, but not consistently (ie: a site may fail with this error while starting up, but then be recycled and start up fine with remote rules in place, then fail with this error again after recycling another day, recycle again and start up fine.) Because it's inconsistent, I don't think it's a problem with the rules, but it's so strange I'm not sure what to think.
The issue is closed..but I don't think it's resolved. Especially if this behavior isn't intended.


Create search field component

Create a search field component.
Design

https://www.figma.com/file/xZ0IW0MJO0vnFicmrHiKaY/Components-Post?type=design&node-id=16739%3A53259&mode=design&t=i6IlqbsNcGrr8I6l-1
Features

Has a placeholder icon that disappears when a search query is entered
Has a clear button that will clear the current search query and focus the input field afterwards
Has an optional suggestion list (which is basically a styleable datalist)

Suggestions can either trigger the search directly (like a link) or just put the value into the input box (specific feature of the suggestionlist component that could also be used in a combobox where selecting a suggestion does not submit a search)
A maximum of 8 items are displayed at any given time (recommendation, use overflow scrolling if there are more)
If the list is empty, it should be hidden
While selecting suggestions, the focus should stay inside the text box in case the user continues to type
Number of characters to trigger the first suggestion get call should be configurable
Suggestion items can have any icon
Text input that matches a part of the suggestion will render this part in bold
Suggestion items can be categorized with headings
The suggestion list can either be rendered inline or as a popover (internet header uses inline rendering as the searchbox itself is already in a popover)

Structure proposal
Floating label
<post-search id="search" floating>
  Search
  <post-suggestion-list slot="suggestions" popover>
    <post-suggestion>Se</suggestion>
  </post-suggestion-list>
</post-search>

Non floating label
Search
<post-search id="search">
  <post-suggestion-list slot="suggestions">
    <post-suggestion>Se</suggestion>
  </post-suggestion-list>
</post-search>

This approach could use the <post-popovercontainer> internally.
<Host>
  <input />
  <post-popovercontainer>
    <slot name="suggestion-list"></slot>
  </post-popovercontainer>
</Host>

Alternative structure proposal (more generic, more complicated):
<post-textinput clearbutton type="search" floating id="search">
  <post-icon name="1000" slot="placeholder-icon"></post-icon>
  Search
</post-textinput>

### Tasks
- [ ] Reach agreement on component naming and structure
- [ ] Create a web component for the search input
- [ ] Create a documentation page
- [ ] Add end to end tests

Initial Design Ticket: https://github.com/swisspost/design-system/issues/1960

Closing because definition and requirements are outdated. Components that will make a suggestion box possible need to be implemented separately.


Add separate page with license and legal clauses GDPR, cookies etc

Is there a target domain for the tutorials (e.g., tutorials.openxr.khronos.org)? Using *.khronos.org may make it easier because Khronos has a legal notice page (https://www.khronos.org/legal/privacy).
A request has been sent to WebServices for general text.

@rbessems awaiting the text for this. Also, what licence should the website source text be under?

In the site footer, please add the following text.
"OpenXR Tutorial is a Khronos owned and operated site. Khronos Legal Notices, Privacy Policy, and Code of Conduct apply.

Done.


chore(release): prepare for 0.28.0

🧀

LGTM - obv, needs to incorporate any changes that get merged in the next little while.


[stable/fluentd] Update Fluentd image, service account annotations and documentation

Is this a new chart
This is an existing chart and it needs updating in this repo as there are no other charts currently available that do everything this one does. With these changes hopefully this chart will be more relevant to the ongoing work on fluent/fluentd (currently only a daemonset).
What this PR does / why we need it:

Update the docker image (both registry and tag) to bring it up to date
Allow for a dynamic command to be run after installing plugins to support other images
Add support for service account annotations which are required for AWS IRSA support
Tidy up documentation

Which issue this PR fixes
n/a
Special notes for your reviewer:
This PR can replace #22842 (open) and #22424 (closed due to inactivity).
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

@hectorj2f could you take a look at this for me?

How about @rendhalver or @miouge1?

Could someone add an OK to test tag please?

Could someone please take a look at this now I've re-based again?
@rendhalver @miouge1 @hectorj2f  @rendhalver @miouge1 @hectorj2f

Not stale.


Feedback from scorecard

What does this PR do?
This PR implements the feedback from scorecard sheet
Description of Task to be completed?
The task implements ability for admin user to manage users (create, edit, delete and search). It also implements profile view and edit of logged in user

Proper validation message when trying to create documents without filling all fields properly
Move the pagination limit to the top right above the list and also the pagination navigation bottom right
Write client side test for actions

How should this be manually tested?

Visit e-docman.herokuapp.com
Clone the repo using : git clone https://github.com/andela-eshaibu/document_management_system.git
Run the following command in your terminal to install dependencies

cd document_management_system
npm install

Create a .env file in the repo's root directory, copy the content of the .env.sample file and  paste in .env file.
Edit the .env file content with the appropriate value for your database url values and default information about the default admin.
Run npm start to start application
Open your browser and navigate to http://localhost:8000

Any background context you want to provide?
N/A
Screenshots (if appropriate)
N/A

Coverage decreased (-0.003%) to 93.533% when pulling<PHONE_NUMBER>3c1d3c96b9e3c4826cd17e51dd85a5 on feedback into a4dc6b0ac8277c271b9efecc3139cab660d73c9c on develop.


Example Site: Unable to run 'bundle exec jekyll serve'

I tried to follow the readme instructions to have a go at the example site but I got this when running the command in subject.
ERROR: YOUR SITE COULD NOT BE BUILT:                     ------------------------------------                     Key related_posts cannot be set in the drop.

Hi @p2635 👋 Can you give me some more information and maybe share your _config.yml and explain a bit about how you are organizing your files?


Fix/vaccinecert 1626 refactoring who component for recovery cert

During the refactoring of the certificate forms with the new who-checkbox component, the recovery form was left out. This was redone and this change also brings the red border for the checkbox with it.

The code looks fine, but we still have issues related to the testing suite.


Disable generic type recognition option.

example:
Get.put(GeneralFragmentInAppStagePageLogic<TFFragmentContentDynamicState, TFFragmentContent>());

// when "<TFFragmentContentDynamicState, TFFragmentContent>" is unknown
Get.delete<GeneralFragmentInAppStagePageLogic>();

Get.put(GeneralFragmentInAppStagePageLogic<TFFragmentContentDynamicState, TFFragmentContent>());

but:
[GETX] Instance "GeneralFragmentInAppStagePageLogic<TFFragmentContentDynamicState, TFFragmentContent>" has been created
[GETX] Instance "GeneralFragmentInAppStagePageLogic<TFFragmentContentDynamicState, TFFragmentContent>" has been initialized

[GETX] Instance "GeneralFragmentInAppStagePageLogic<DynamicState, BaseFragmentContent<DynamicState>>" already removed.

[GETX] Instance "GeneralFragmentInAppStagePageLogic<TFFragmentContentDynamicState, TFFragmentContent>" has been created
[GETX] Instance "GeneralFragmentInAppStagePageLogic<TFFragmentContentDynamicState, TFFragmentContent>" has been initialized

TFFragmentContentDynamicState extent DynamicState
TFFragmentContent extent BaseFragmentContent<TFFragmentContentDynamicState >

maybe this solution will solve your problem click


add ara image license

Adding licence on ARA picture.

@lusoalex
Hello !
Thanks for your PR ! :D
Do you mind if we change the target branch from master to version/4.0.0 due to the fact that this PR isn't a Release's hotfix ?

@lusoalex
Hello !
Thanks for your PR ! :D
Do you mind if we change the target branch from master to version/4.0.0 due to the fact that this PR isn't a Release's hotfix ?

No for sure, I do it right now ;)


feat(select): js scrollbar replaced with css

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

36 of 36 (100.0%)  changed or added relevant lines in 6 files are covered.
14 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.1%) to 79.553%

Files with Coverage Reduction
New Missed Lines
%

packages/scrollbar/src/Component.tsx
14
65.14%

Totals

Change from base Build<PHONE_NUMBER>:
-0.1%

Covered Lines:
8100

Relevant Lines:
9210

💛  - Coveralls

Припарковано, ждёт @Oladii


Make peer dependencies more lenient.

From v7 onwards, NPM tries to automatically install peer dependencies and will throw, if any packages in the dependency tree have conflicting versions of the same peer dependency [1]. Therefore I suggest to impose soft peer dependency requirements:

react and react-dom greater than  16.8 (the version which introduced hooks)
react-window greater than 1.8.5 (the version this lib is tested against).

I'd also remove the peer dependency on the typings, as that might throw off non-Typescript users.
[1] https://github.blog/2020-10-13-presenting-v7-0-0-of-the-npm-cli/

Hmm, what about making the typings optional then?

Optional would work, sure. But in this case also, I'd go with a minimal required version rather than a specific one.


Hikari count connection as active even connection is stoped

Hi, i found issues after using:
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle in transaction'
-- with some filters

After some time (about 10-20 time pg_terminate_backend  ) hikari count terminated connection as active.
As result it can have half of pool size those connections and only restart can help
F.e. screen from jprofiler hikari mbean info - 8 active connection

but in db only 5 connection
select application_name,
pid,
state,
client_addr
from pg_stat_activity
where application_name like '%local_READ_O%'
order by pid desc;
I suppose such situation can be happened during
------------------------------------------------------------------- connection
----transaction--------1point--|---2point-------------
if pg_terminate_backend are called at:

point 1 then tx manager throws exception and hikari process connection closing successfully
point 2 then hikari do not catch it

I think it could help to know the versions of

HikariCP
Postgres JDBC driver (assuming you talk about Postgres)
Postgres Server

And also to have your HibariCP config helps for discussions.

versions:

HikariCP 4.0.3  also checked on last version (6.0.0) - the same behaviour
org.postgresql:postgresql 42.3.8
Postgres 14.8 (Ubuntu 14.8-1.pgdg20.04+1)

@Bean("dataSource")
    public DataSource dataSource(
        HikariConfig hikariConfig,
        @Value("${spring.datasource.writeParams:?targetServerType=master}") String writeParams,
        @Value("${spring.datasource.readParams:?targetServerType=preferSlave&loadBalanceHosts=true}") String readParams,
        @Value("${spring.application.name}") String applicationName) {
        AbstractRoutingDataSource routingDataSource = new AbstractRoutingDataSource() {

            @Override
            protected Object determineCurrentLookupKey() {
                return TransactionSynchronizationManager.isCurrentTransactionReadOnly()
                    ? DataSourceType.READ_ONLY
                    : DataSourceType.READ_WRITE;
            }
        };

        HikariConfigMutator mutator = new HikariConfigMutator(hikariConfig);

        Map<Object, Object> dataSourceMap = new HashMap<>();
        dataSourceMap.put(
            DataSourceType.READ_WRITE,
            new HikariDataSourceWrapper(mutator.mutateDataSource(writeParams, applicationName, DataSourceType.READ_WRITE))
        );

        dataSourceMap.put(
            DataSourceType.READ_ONLY,
            new HikariDataSourceWrapper(mutator.mutateDataSource(readParams, applicationName, DataSourceType.READ_ONLY))
        );

        routingDataSource.setTargetDataSources(dataSourceMap);
        return routingDataSource;
    }

Bean
    public HikariConfig hikariConfig(
        @Value("${database.driver}") String jdbcDriver,
        @Value("${database.dbName}") String dbName,
        @Value("${database.username}") String username,
        @Value("${database.password}") String password,
        @Value("${database.maxPoolSize:20}") Integer maxPoolSize,
        @Value("${database.minPoolSize:20}") Integer minIdle,
        @Value("${database.connectionTimeout:15000}") Long connectionTimeout,
        @Value("${database.maxLifeTime:900000}") Long maxLifeTime,
        MemberHostExtractor memberHostExtractor
    ) throws ClassNotFoundException {
        // load postgresql driver, needed for tracing
        Class.forName("org.postgresql.Driver");

        HikariConfig config = new HikariConfig();
        config.setUsername(username);
        config.setJdbcUrl(buildJdbcUrl(memberHostExtractor, dbName));
        config.setPassword(password);
        config.setDriverClassName(jdbcDriver);
        config.setMaximumPoolSize(maxPoolSize / DATASOURCE_COUNT);
        config.setMinimumIdle(minIdle / DATASOURCE_COUNT);
        config.setConnectionTimeout(connectionTimeout);
        config.setLeakDetectionThreshold(60_000);
        config.setMaxLifetime(maxLifeTime);
        config.setAutoCommit(false);
        config.setRegisterMbeans(true);

        return config;
    }

@DmitryShabalin Have you read the important note on the HikariCP Github Main page about the usage of tcpKeepAlive?
The referred article from Cybertec shows how to use this to detect dead connections. Which might help with your problem of counting active connections.

Ensure PostgreSQL driver socketTimeout is set, and as mentioned by @svendiedrichsen ensure that keepAlive is configured.


About the preprocessing method on the cityscapes data

Hello, thanks for providing this fantastic method!
When tracing the code, I found a comment mentioning "use random crop, don't resize @ cityscapes," but the program resizes the image in the following snippets but doesn't random crop it. Is it just a typo?
Thanks a lot!

Hey Kevin, thanks for your comment!
You are right, in deed an old comment. I fixed the documentation to the current implementation.
Thanks,
Shahaf


add create get and list on app insight

Hey,
you should reformat with yapf ``yapf -ir .`
Do you need additional steps to link the app insight to a static webapp ? Just give the instrumentation key to the webapp config is sufficient ?


Add coverall coverage report and badge

We should report test coverage for the whole monorepo, including a badge in the README.

@jeromesimeon  is there any available badge that we can use for test coverage? I think we already have test coverage in travis build

@Niladri24dutta You can find test coverage badge examples in some of the other projects. e.g., https://github.com/accordproject/cicero/blob/master/README.md
The issue here is that I don't think we are reporting those properly to coverall. Also the cicero-cli does not have code coverage in place.


The problem is really the benchmark of it.
The key part is marshalling and unmarshalling, so gRPC itself operations!
With the benchmark we did.... we have ZERO of those operations, so you only measure splitting process part not the benefit of it (:
So we have a few options:

Create better micro benchmarks, one that involve actually some network traffic (not to test network latency, but rather marshalling /umarshalling HUGE responses).
Create better e2e benchmarks. So something on top of e2e framework or also https://github.com/thanos-io/thanosbench with some e2e tests
Just don't benchmark here. Set an optional flag for splitting. THIS IS recommended for a reason by gRPC / protobuf guys so I assume there will be a benefit in real-life setups. With enabling this as optional tweak users can tweak it on production setups and see if it helps :hugs:

I think we can do all those things.. But in this PR we can focus on third item, so flag, then focus on more benchmarking if you have time!

Just got pinged. I think the next think is to actually improve benchmark (:

Don't think I have time to continue this now.


Generated fonts not appearing properly in Windows

Hello all,
I'm facing a kind of bizarre issue with some svg icons, after generating fonts, they'd appear as expected under Mac, but totally wrong in windows.
Example with this circle alert icon:
alert-round.svg.zip
Under Mac:

Under Windows:

I used SVGO to cleanup all my icons, but not solving the issue.
Note that this doesn't happen with all the icons, some of them would render just perfect
Thanks in advance

Try this
alert-round.svg-2.zip

Thank you @tancredi  this worked perfect,
May you explain what happened :) , why is this happening only on Windows?
I have several of them giving the same issue

Not sure about windows, just opened the SVG in Illustrator and fixed it up as it displayed like so:

May be the use of fill-rule="evenodd" but I'm really not that versed in SVG specifications

oh ok thanks anyway for pointing that out

Fixed by reworking the SVGs to avoid evenodd filling rule.
This is a common problem preventing Fonts and Android svg rendering from working properly
There is this awesome plugin that helps converting from even odd to non-zero
https://twitter.com/evanwallace/status/1189594907989565440


Update user guides and docs regarding tasks using Java toolchains

This PR updates javadocs and user guides to include the expanded role of toolchains in the core Java tasks.
This is a follow-up for:

https://github.com/gradle/gradle/pull/22108

I applied all the grammar and wording changes. I think they all make sense.

Thanks for the review, @tresat ! I applied your suggestions.
@bot-gradle test and merge

Your PR is queued. See the queue page for details.

OK, I've already triggered a build for you.


Verify this action in GitHub Marketplace

Depending on a Github organization's security settings, access to actions may be restricted to ones that are "verified in the GitHub Marketplace". Otherwise the workflow exits with a message like:
Error: .github#L1
erlef/setup-beam@v1 is not allowed to be used in MyOrg/my-repo. Actions in this workflow must be: within a repository owned by MyOrg, created by GitHub, or verified in the GitHub Marketplace.

Would it be possible to get this action verified?

Related: https://github.com/erlef/setup-beam/issues/129
I'm still not sure how we'd go about with a good process for this. I'm Ok with 2FA, though.
@starbelly, thoughts?

@paulo-ferraz-oliveira Oh, indeed we should get verified. I'll look into this tomorrow.

We need to:

check constraints for this, like the 2FA thing, for example
potentially a way to automate the release or, at the very least, have a process for it

(I'll keep following the thread)

For future ref: https://docs.github.com/en/apps/github-marketplace/github-marketplace-overview/about-marketplace-badges
There are bullet points there that we'd need to discuss on how to move forward.

For future ref: https://docs.github.com/en/apps/github-marketplace/github-marketplace-overview/about-marketplace-badges
There are bullet points there that we'd need to discuss on how to move forward.

Verified ownership of their domain and has a verified badge on their profile
Confirmed their email address so GitHub Support can reach the organization
Required two-factor authentication for their organization.

👆Anyone against implementing this? Seems like low-hanging-fruit IMHO. Would be great to remove this hurdle to BEAM adoption in corporate environments.

Thanks for the feedback. Yes, I wasn't aware that the account ownership is unclear.
In general, verifying a GitHub organization is dead simple if you are the account owner and control the domain that you list on the organization's page (for https://github.com/erlef it is https://erlef.org/). It's just a matter of adding a DNS record.
Having a confirmed email address is also not something I would describe as difficult, if the organization's emails are working.
So the only remaining question is about enabling Two-Factor Authentication, which to me seems perfectly natural for the type of account that we're talking about. Yes, it would affect the list of members at https://github.com/orgs/erlef/people, but only those who do not already have 2FA activated on their personal accounts.
Either way... it affects anyone trying to set up a CI/CD pipeline for the first time at a company / org that has the security setting enabled to "restrict actions to ones that are verified in the GitHub Marketplace". From what I can see, the setting can only be flipped for the whole org, affecting all of the repos, which would probably be a no-go in most corporate environments. So my feeling is that not having this action is probably a barrier-to-entry for proof-of-concept / demo type work at bigger organizations, which is the scenario that I was in when raising the issue.

P.S. In case it helps, here is a run-through that shows where the relevant settings can be found: https://ludwiguer.medium.com/add-a-verified-badge-to-your-github-organization-41391834a16a

Hi all, we are in the process of verification now 🎉 The final verification request has been sent to github so merely waiting on their response 😄

Awesome! Thanks for that 😁

The erlef org. is now Verified (https://github.com/erlef). That's a step in the right direction 😄

I tried publishing a new version of this to the Marketplace to see if it'd show as "creator-verified" but it didn't. I'm not sure there's a job running to identify this, or something else, but we might be missing some more actions, @starbelly.
On the other hand, @petrus-jvrensburg, could you run this under the initial conditions that got you to create the issue, and tell us how it went? (the doc. seems to indicate "Verified" in org. is different from "Verified" in action, but I tested this in another org. and I got ✅)

@paulo-ferraz-oliveira The domain is verified, but we are still waiting on overall verification which is a request to github. Presumably this is a manual job, as such I would not expect it to be complete until Monday.

To note we are still waiting, I suppose if I don't hear back by tomorrow, I will send an email.

Emailed :)

Got response, next step which I've asked someone else to fill out is a form required for github tech partners.

Good ol' human trust!

The process for becoming a tech. partner (which is required to for actions to be verified) has started.

Good'ol bureaucracy in the works...

@starbelly, did this ever move forward? Are we waiting for stuff on ErlEF's end or GitHub's?

@starbelly, did this ever move forward? Are we waiting for stuff on ErlEF's end or GitHub's?

@paulo-ferraz-oliveira No, but thanks for the ping. I need to chase someone down.


chore(lobbies): disable fetch state by default

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#583  👈
#582
#580
#579
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @NathanFlurry and the rest of your teammates on  Graphite


Module not found: Error: Can't resolve 'react' in

Try run the examples/basic app4,it throw error as :
ERROR in ./src/Button.tsx 1:0-31
Module not found: Error: Can't resolve 'react' in '/Users/xxx/Downloads/fronts-master/examples/basic/app4/src'
@ ./src/App.tsx 2:0-35 4:152-163
@ ./src/bootstrap.tsx 4:0-24 6:52-55
@ ./src/index.ts 1:0-21
ERROR in ./src/bootstrap.tsx 1:0-26
Module not found: Error: Can't resolve 'react' in '/Users/xxx/Downloads/fronts-master/examples/basic/app4/src'
@ ./src/index.ts 1:0-21
ERROR in ./src/bootstrap.tsx 3:0-36
Module not found: Error: Can't resolve 'fronts-react' in '/Users/xxx/Downloads/fronts-master/examples/basic/app4/src'
@ ./src/index.ts 1:0-21
ERROR in ../node_modules/react-dom/cjs/react-dom.development.js 18:12-28
Module not found: Error: Can't resolve 'react' in '/Users/xxx/Downloads/fronts-master/examples/basic/node_modules/react-dom/cjs'
@ ../node_modules/react-dom/index.js 37:2-60
@ ./src/bootstrap.tsx 2:0-33 6:2-17 8:4-35
@ ./src/index.ts 1:0-21
What need to do to make example/basic works,thank you.

Hi @shuyue426 , you must execute yarn build in the root directory, and cd example/basic && yarn install.

it works,thanks.


Update HmacKey documentation when cloud.google.com links are up.

Since we are implementing a non-GA feature the documentation links will not be correct for a while.

Removing from the GA milestone because the feature is not going to be GA on time.


Mocking msalInstance for testing

This issue is for a

[ ] bug report -> please search issues before submitting
[ ] feature request
[x] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Hi,is there a recommended/documented approach to mocking the MSAL instance used by the provider when writing unit tests?
e.g. If one was to write a minimal unit test of the <MainContent /> component in app.jsx that just tested whether the correct component was shown based on the authentication status. What would that look like?
Thanks in advance!

FWIW/for anyone else reading, the easiest way I've found so far is just injecting the MsalContext with whatever you want. e.g. when rendering commponents using something like react-testing-library you could use something like this:
    const mockActiveAccount = {
      tenantId: environment.msalConfig.auth.tenantId,
      idTokenClaims: {
        aud: environment.msalConfig.auth.clientId,
        roles: ['Admin'],
      },
    } as AccountInfo;
    const mockGetActiveAccount = jest.fn(() => mockActiveAccount);
    render(
      <MsalContext.Provider
        value={{
          instance: {
            getActiveAccount: mockGetActiveAccount,
          },
          inProgress: InteractionStatus.None,
          accounts: [mockActiveAccount],
          logger: jest.fn() as unknown as Logger,
        }}
      >
        <YourComponent />
      </MsalContext.Provider>
    );

The above will be enough to test most of the hooks (e.g. useIsAuthenticated, useMsal etc) as they all rely on the context.
In the above example, for instance, useIsAuthenticated would return true by virtue of the accounts property having a value in the array. If you wanted to simulate an unauthenticated user, you could just use an empty array for accounts instead.

@ceuk closing issue due to inactivity.


Ensure pair callback data is set to null when it's null.

Master branch version of #39504.
Fixes #39487
Fixes #39500

Thanks!


Add Set terminate Option for user

Add an option to Terminate the current session and the session remains in the terminated state until the user calls the unset terminate option.

Do we intend to expose this in other language bindings?

Since we wanted to add runoptions instead of individual APIs, created a new PR for this work https://github.com/microsoft/onnxruntime-genai/pull/1011. Will close this once the PR is merged.


Popup descriptions hidden in edit page

version: commit 5e7eb03
Steps to reproduce:

Go to edit VM page
Click at "Cluster" label

Actual result:
Only small arrow is shown instead of whole tootlip bubble.

Fixed by #603


Update rust dependencies

Update outdated rust dependencies.

@mmastrac There is an outdated dep in conn-pool:
conn-pool
================
Name               Project  Compat  Latest  Kind         Platform
----               -------  ------  ------  ----         --------
genetic_algorithm  0.7.2    ---     0.10.2  Development  ---

Is this intentional?

@mmastrac There is an outdated dep in conn-pool:
conn-pool
================
Name               Project  Compat  Latest  Kind         Platform
----               -------  ------  ------  ----         --------
genetic_algorithm  0.7.2    ---     0.10.2  Development  ---

Is this intentional?

Yeah, it was the latest at the time but I worked with the author to make some breaking API changes -- the upgrade will land as part of https://github.com/edgedb/edgedb/pull/7681


Minor fixes

There is an issue if you set the date from outside. The .day-item doesn't get the selected class.
And if you are in vertical state and you change the item from outside, it doesn't get updated.
To prove all this, I've modified the demo page to demonstrate the fixes.

Thanks! This should work as of now in master.


Downloading a file inside an Extension

Hi, I am developing a photo editor https://www.Photopea.com
I decided to make it a Brackets extension. You can find and install it by searching for "photopea" in an extension manager.
I have noticed, that files can not be saved (File - Save as PSD does nothing). My editor saves files through a "download" attribute of an anchor tag.
XY

It executes a new MouseEvent on this tag. This works in all browsers, but it seems it does not work in Brackets. Are you going to support this in an Extension environment any time soon?
It also seems like you don't support a filter-invert of CSS styles (icons in Photopea should be gray, not black).
filter: invert(  0.2  );

Keep in mind that ATM Brackets is build on top of Chrome 49 on Windows and macOS and Chrome 29 on Linux.
So for the filter attribute you probabily need to add the -webkit prefix.
For the MouseEvent we should know more. For example how did you create Save as PSD?

Chrome is updated like every month. Why do you use 1.5 years and 3 years old versions? Is it because you haven't released a new Brackets version for Linux since 2013?
I am saving files this way:
var data = new Uint8Array(arrayBuffer);
var a = document.createElement( "a" );
var blob = new Blob([data]);
var url = window.URL.createObjectURL( blob );
a.href = url;
a.download = "file.psd";
document.body.appendChild(a);
a.click();
document.body.removeChild(a);

I tryed that snippet on Brackets console (Debug \ Show Developer Tools) replacing arrayBuffer with [].
I see a  tag created and Brackets ask me to download the file. It seems to work to me.
Maybe some other @adobe/brackets-committers can help more here.
Just FYI updating CEF (which is the Chrome based software used by Brackets) is a big task, usually done by Adobe. I don't know if there is any change to update it (or switch the shell).
Brackets is divided in two parts: the shell code and the browser one. It is possible to release newer versions of the browser code without touching the shell. So Linux builds are updated every release.


panic on start

I've done git pull cargo run --release
after that it panics, here is debug run
building on macos amd64
➜  ➜  barnes-hut git:(master) RUST_BACKTRACE=full ./target/debug/barnes-hut
thread 'main' panicked at /Users/mag1/.cargo/git/checkouts/quarkstrom-59b32b829dff6ef4/8aa27db/src/lib.rs:152:14:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0:        0x104a22563 - std::backtrace_rs::backtrace::libunwind::trace::h3e5b2c5356eac916
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:        0x104a22563 - std::backtrace_rs::backtrace::trace_unsynchronized::h0d222ed0e6a20102
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x104a22563 - std::sys_common::backtrace::_print_fmt::h0b309fc5cd20c59f
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/sys_common/backtrace.rs:68:5
   3:        0x104a22563 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h31f9e6a79997ef5f
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x104a4223b - core::fmt::rt::Argument::fmt::h40b9b7c1e9bb0f98
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/fmt/rt.rs:165:63
   5:        0x104a4223b - core::fmt::write::h85020c7a61eba8ce
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/fmt/mod.rs:1168:21
   6:        0x104a2038e - std::io::Write::write_fmt::hcc19df4fa4e73a55
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/io/mod.rs:1835:15
   7:        0x104a22351 - std::sys_common::backtrace::_print::h9fe487822a10be3b
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/sys_common/backtrace.rs:47:5
   8:        0x104a22351 - std::sys_common::backtrace::print::h45d16fbbed4d1f36
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/sys_common/backtrace.rs:34:9
   9:        0x104a235f9 - std::panicking::default_hook::{{closure}}::h885e10af905b3914
  10:        0x104a23378 - std::panicking::default_hook::hbcf94856b92e65ef
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:298:9
  11:        0x104a24016 - std::panicking::rust_panic_with_hook::h9e15a0c845c470d6
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:795:13
  12:        0x104a2394a - std::panicking::begin_panic_handler::{{closure}}::h3cd5140e29c8cfc1
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:656:13
  13:        0x104a22a39 - std::sys_common::backtrace::__rust_end_short_backtrace::h124259ee4cbc9197
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/sys_common/backtrace.rs:171:18
  14:        0x104a236b6 - rust_begin_unwind
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:652:5
  15:        0x104a7f382 - core::panicking::panic_fmt::h9772381b80805b71
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/panicking.rs:72:14
  16:        0x104a7f444 - core::panicking::panic::h2a3e12572053020c
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/panicking.rs:146:5
  17:        0x104a7f2e8 - core::option::unwrap_failed::hf02468572e4a732f
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/option.rs:1985:5
  18:        0x104072260 - core::option::Option<T>::unwrap::hcf7fc7af46cf7ed3
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/option.rs:935:21
  19:        0x104072260 - quarkstrom::State::new::{{closure}}::h0fb164576804ce7b
                               at /Users/mag1/.cargo/git/checkouts/quarkstrom-59b32b829dff6ef4/8aa27db/src/lib.rs:145:23
  20:        0x10406acd1 - pollster::block_on::h5138fa9fb06a4cdd
                               at /Users/mag1/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pollster-0.3.0/src/lib.rs:128:15
  21:        0x10407147c - quarkstrom::run::h22b6dc877f531568
                               at /Users/mag1/.cargo/git/checkouts/quarkstrom-59b32b829dff6ef4/8aa27db/src/lib.rs:634:21
  22:        0x10408e2ad - barnes_hut::main::h45cfb77960f899f2
                               at /Users/mag1/dev_tmp/rusting/barnes-hut/src/main.rs:30:5
  23:        0x1040ba1fe - core::ops::function::FnOnce::call_once::h22ebe57cfef39855
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/ops/function.rs:250:5
  24:        0x10407d1b1 - std::sys_common::backtrace::__rust_begin_short_backtrace::hbebc4ccbf12a881a
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/sys_common/backtrace.rs:155:18
  25:        0x1040b4ee4 - std::rt::lang_start::{{closure}}::h103b26ee385fea99
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/rt.rs:159:18
  26:        0x104a1dd07 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hde6295cc14fa5e33
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/ops/function.rs:284:13
  27:        0x104a1dd07 - std::panicking::try::do_call::h7ac29dd06f7ef49a
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:559:40
  28:        0x104a1dd07 - std::panicking::try::hc51707793193dcd5
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:523:19
  29:        0x104a1dd07 - std::panic::catch_unwind::h0e679fc68f6830e5
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panic.rs:149:14
  30:        0x104a1dd07 - std::rt::lang_start_internal::{{closure}}::ha13f2fee1223fb5c
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/rt.rs:141:48
  31:        0x104a1dd07 - std::panicking::try::do_call::h1f7f85fb8df70a9c
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:559:40
  32:        0x104a1dd07 - std::panicking::try::hb8aefacb3538256f
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:523:19
  33:        0x104a1dd07 - std::panic::catch_unwind::hc6b8292b163c9a2e
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panic.rs:149:14
  34:        0x104a1dd07 - std::rt::lang_start_internal::h0c6931bcd088f2f1
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/rt.rs:141:20
  35:        0x1040b4eb7 - std::rt::lang_start::hdbe1865eb8f115f8
                               at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/rt.rs:158:17
  36:        0x10408e668 - _main
  37:     0x7ff80e932345 - <unknown>

my bad https://github.com/DeadlockCode/n-body/issues/1 that is a way to patch

I sadly don't have a MacOS machine so I can't test this myself and I don't really know anything about MacOS, but it's panicing when it's trying to get a physical device with Vulkan backend. According to some web searches, MacOS doesn't have native Vulkan support but that might be fixable by either a) installing MoltenVK or b) switching to Metal backend when on MacOS. If installing MoltenVK works then I'll consider this fixed but if not I'd like to know if changing this line to
backends: wgpu::Backends::VULKAN | backends: wgpu::Backends::METAL,

solves the issue.

I have molten-vk installed - didnt work
i've patched quarkstrom in my fork - didnt work either
Will try to look into searches as well

@DeadlockCode changing ln135 does not solve the issue:
cargo run --release
   Compiling quarkstrom v0.1.0 (/Volumes/T7W01/Downloads-T7/tmp/quarkstrom)
    Finished `release` profile [optimized] target(s) in 0.96s
     Running `target/release/quarkstrom`
thread 'main' panicked at /Volumes/T7W01/Downloads-T7/tmp/quarkstrom/src/lib.rs:170:14:
called `Result::unwrap()` on an `Err` value: RequestDeviceError
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Ok Ive made it able to run https://github.com/DeadlockCode/quarkstrom/compare/master...umag:quarkstrom:master but its slow

@umag Thanks. It works!!


Cloning host broken

Rancher versions: 5/23
Steps to Reproduce:

Try to clone a host

Results:  Get an error page - ember.prod.js:16699 TypeError: Cannot read property 'find' of undefined

#8832