main page

track

Use DeepSpeed to do RLHF
Installation
Python version 3.7

## example requirements
git clone https://github.com/microsoft/DeepSpeedExamples
cd DeepSpeedExamples/applications/DeepSpeed-Chat
pip install -r requirements.txt  [exclude torch transformers deepspeed]

## torch/tf with cuda 11.3
pip install tensorflow==2.8.0
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113
pip install torchtext==0.13.0

## lastest huggingface
pip install git+https://github.com/huggingface/transformers
-> Successfully installed transformers-4.29.0.dev0

## deepspeed
git clone https://github.com/microsoft/DeepSpeed
git checkout v0.9.0
pip install .
-> Successfully installed deepspeed-0.9.0+0b5252bb

ds_report

Example 1: Coffee Time Training for a 1.3B ChatGPT Model
python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_gpu

## requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='cdn-lfs.huggingface.co', port=443): Read timed out. (read timeout=10.0)
## download model
from transformers import AutoModel
model = AutoModel.from_pretrained('facebook/opt-1.3b')

or use snapshot_download from https://huggingface.co/docs/huggingface_hub/v0.14.0.rc0/guides/download#download-an-entire-repository

## ConnectionError: Couldn't reach https://huggingface.co/datasets/Dahoas/rm-static/resolve/64fd53cc91f7cb73b283a6e4f661205e277d23c9/data/test-00000-of-00001-8c7c51afc6d45980.parquet (ConnectionError(ProtocolError('Connection aborted.', OSError(0, 'Error'))))
## download data
from datasets import load_dataset
load_dataset("Dahoas/full-hh-rlhf")

## OPT-1.3B on V100(32GB)
Step1: 60min
Step2: 17min
Step3: 2h19min

Dataset

Anthropic's Helpful Harmless dataset:

Dahoas/rm-static
Dahoas/full-hh-rlhf
Dahoas/synthetic-instruct-gptj-pairwise

Other comparison dataset

yitingxie/rlhf-reward-datasets
openai/webgpt_comparisons
stanfordnlp/SHP

Plain Alpaca-LoRA to do SFT
## Preparing
pip install -r requirements.txt

## download model timeout
from huggingface_hub import snapshot_download
snapshot_download(repo_id="decapoda-research/llama-7b-hf")
from datasets import load_dataset
load_dataset("yahma/alpaca-cleaned")

ref: https://huggingface.co/docs/huggingface_hub/v0.14.0.rc0/guides/download#download-an-entire-repository

## model too big to save docker image
save ~/.cache/huggingface somewhere, and soft link folder to ~/.cache/huggingface

Providing valuable insights into the latest models, including number of parameters, fine-tuning datasets and techniques, and hardware specifications.
Practical guides for LLM alignment post-training, include dataset, benchmark datasets, efficient training libraries and techniques; also involves history of pre-trained LLM.

Pre-trained Base Models
 Simple Version

OpenAI: GPT-1, GPT-2, GPT-3, InstructGPT, Code-davinci-002, GPT-3.5, GPT-4(-8k/32k)
Anthropic: Claude-v1, Claude Instant
Meta: OPT, Galactica, LLaMA
huggingface BigScience: BLOOM (176B), BLOOMZ, mT0
EleutherAI: GPT-Neo, GPT-J (6B), GPT-NeoX (20B), Pythia
TogetherCompute: GPT-JT, RedPajama-7B, RedPajama-INCITE
Berkeley: OpenLLaMA
MosaicML: MPT-7B, MPT-7B-Instruct/Chat
TII: Falcon-7/40B-(instruct)
BlinkDL: RWKV-4-Pile, RWKV-4-PilePlus
Tsinghua THUDM: GLM-130B, ChatGLM-6B
Cerebras: Cerebras-GPT
Google: T5, mT5, LaMDA, Pathways, PaLM, UL2, Flan-T5, Flan-UL2, Bard, PaLM-E, PaLM 2, MoE, Switch Transformer, GLaM, ST-MoE, MoE Routing
DeepMind: Gopher, Chinchilla, Sparrow
Nvidia: Megatron-Turing NLG (530B)

OpenAI

2018/06, GPT-1 (117m)
2019/02, GPT-2 (1.5B)
2020/06, GPT-3 (175B): ada(350M), babbage(1.3B), curie(6.7B), davinci(175B), detail here
2022/01, InstructGPT-3: text-ada(350M), text-babbage(1.3B), text-curie(6.7B), text-davinci-001(175B)
2022/02, Code-davinci-002
GPT-3.5 (175B): text-davinci-002 (2022/03), text-davinci-003 (2022/11), ChatGPT (2022/11), gpt-3.5-turbo (2023/03)
2023/03, GPT-4(-8k/32k)

Anthropic

Claude-v1: 2023/03, state-of-the-art high-performance model, context window 9k/100k tokens
Claude Instant: 2023/03, lighter, less expensive, and much faster option, context window 9k/100k tokens

Meta

OPT (125M/350M/1.3B/2.7B/6.7B/13B/30B/66B/175B): 2022/03, pre-trained on (datasets used in RoBERTa, the Pile, PushShift.io Reddit) using metaseq, 1/7th the carbon footprint if GPT-3, combining Meta’s open source Fully Sharded Data Parallel (FSDP) API and NVIDIA’s tensor parallel abstraction within Megatron-LM, contain predominantly English text and a small amount of non-English data via CommonCrawl, released under a noncommercial license.
Galactica (125M/1.3B/6.7B/30B/120B): 2022/11, facebook/galactica models are designed to perform scientific tasks, include prompts in pre-training alongside the general corpora, under a non-commercial CC BY-NC 4.0 license
LLaMA (7B/13B/33B/65B): 2023/02, trained LLaMA 65B/33B on 1.4 trillion tokens, LLaMA 7B on one trillion tokens, chose text from the 20 languages with the most speakers, leaked,  under a non-commercial GPL-3.0 license.

huggingface BigScience

BLOOM (176B): 2022/07/11, a multilingual LLM trained on ROOTS corpus (a composite collection of 498 Hugging Face datasets), using 250k vocabulary sizes, seq-len 2048, smaller size model search here, release under commercial friendly BigScience Responsible AI License.
BLOOMZ & mT0: 2022/11, finetune BLOOM & mT5 on our crosslingual task instruction following mixture (xP3), released under commercial friendly bigscience-bloom-rail-1.0 License.

EleutherAI

The Pile: 2020/12/31, a 300B (deduplicated 207B) token open source English-only language modelling dataset, download here.
GPT-Neo (125M/1.3B/2.7B)(Deprecated): 2021/03/21, A set of decoder-only LLMs trained on the Pile, MIT license.
GPT-J (6B): 2021/06/04, EleutherAI/gpt-j-6b, English language model trained on the Pile using mesh-transformer-jax library, seq-len 2048, Apache-2.0 license.
GPT-NeoX (20B): 2022/02/10, EleutherAI/gpt-neox-20b, English language model trained on the Pile using GPT-NeoX library, seq-len 2048, Apache-2.0 license.
Pythia (70M/160M/410M/1B/1.4B/2.8B/6.9B/12B): 2023/02/13, a suite of 8 model sizes on 2 different datasets: the Pile, the Pile deduplication, using gpt-neox library, seq-len 2048, Apache-2.0 license.

TogetherCompute

GPT-JT (6B): 2022/11/29, A fork of GPT-J-6B, fine-tuned on 3.53 billion tokens with open-source dataset and techniques, outperforms most 100B+ parameter models at classification.
RedPajama-Pythia-7B: 2023/04/17, release RedPajama-Data-1T for reproducing "LLaMA" foundation models in a fully open-source way; 40% RedPajama-Data-1T trained RedPajama-Pythia-7B beat Pythia-7B trained on the Pile and StableLM-7B with higher HELM score, still weaker than LLaMA-7B for now; detail see blog1, blog2 and Card.
OpenChatKit: 2023/03/10, fine-tuned for chat from EleutherAI’s GPT-NeoX-20B with over OIG-43M instructions dataset; contributing to a growing corpus of open instruction following dataset.
RedPajama-INCITE (3B/7B): 2023/05/05, open-source 3B model (base/chat/instruct) trained on 800B tokens and finetuned, the strongest model in it’s class and brings LLM to a wide variety of hardware; 80% (800B) trained 7B model beat same class GPT-J/Pythia/LLaMA on HELM and lm-evaluation-harness; releasing RedPajama v2 with 2T Tokens (mix the Pile dataset into RedPajama, more code like the Stack); Apache 2.0 license.
Berkeley/OpenLLaMA: open source reproduction of Meta AI’s LLaMA 7B/3B trained on the RedPajama dataset, provide PyTorch and JAX weights, Apache-2.0 license.

MosaicML

MPT (MosaicML Pretrained Transformer, 7B(6.7B)): 2023/05/05, a GPT-style decoder-only transformers trained from scratch on 1T tokens of text and code (RedPajama, mC4, C4, the Stack Dedup, Semantic Scholar ORC) in 9.5 days at a cost of ~$200k, ALiBi (handle 65k long input) and other optimized techniques, matches the quality of LLaMA-7B; open source for commercial use, Apache-2.0 License.
MPT-7B-Instruct/Chat: finetuning MPT-7B on instruction following dataset and dialogue generation dataset; release mosaicml/dolly_hhrlhf dataset derived from Databricks Dolly-15k and Anthropic’s Helpful and Harmless datasets; CC-By-SA-3.0 (commercially-usable) / CC-By-NC-SA-4.0 (non-commercial use only).

TII (Technology Innovation Institute)

Falcon-7/40B-(instruct): 2023/05/26, pretrained on 1500/1000B tokens of RefinedWeb (apache-2.0) enhanced with curated corpora, finetuned on a mixture of chat/instruct datasets like Baize, come first at huggingface Open LLM Leaderboard at the end of May 2023; commercial use should pay when attributable revenues superior to $1M/year.

BlinkDL

RWKV-4-Pile (169M/430M/1.5B/3B/7B/14B): 2023/04, RWKV: Reinventing RNNs for the Transformer Era, leverages RNN with a linear attention mechanism, trained on the Pile, infinite seq-len, Weights.
RWKV-4-PilePlus (7B/14B): 2023/04, finetuning on [RedPajama + some of Pile v2 = 1.7T tokens].

Tsinghua THUDM

GLM-130B: 2022/10, An Open Bilingual Pre-Trained Model, support english and chinese, trained on 400B text tokens using GLM library, Apache-2.0 license.
ChatGLM-6B: 2023/03, trained with 1T chinese and english tokens, finetuned with instruction following QA and dialogue dataset in chinese language, released under Apache-2.0 license, authorization needed.

Cerebras

Cerebras-GPT: 2023/03, a family of seven GPT models ranging from 111M to 13B, trained Eleuther Pile dataset using the Chinchilla formula, release under the Apache 2.0 license

Google

2019/10/23, T5, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
2020/01/28, LaMDA: blog-2021/03/18, blog-2020/01/28, blog-2022/01/21
2021/10/28, Introducing Pathways: A next-generation AI architecture
2022/04/04, Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance
2022/05/11, Language Models Perform Reasoning via Chain of Thought
2022/10/14, UL2 20B: An Open Source Unified Language Learner
2023/02/01, FLAN: Introducing at 2021/10/06; Flan 2022 Collection for instruction tuning release here; first used in Flan-T5 and Flan-PaLM at 2022/10/20, achieved significant improvements over PaLM; Flan-UL2
2023/02/06, Introducing Bard
2023/03/10, PaLM-E: An embodied multimodal language model.
2023/05/10, Introducing PaLM 2, review the technical report
MoE: Sparsely-Gated Mixture-of-Experts layer (MoE) 2017/01, Switch Transformer 2021/01, GLaM 2021/12, ST-MoE 2022/02, MoE Routing 2022/11

DeepMind

2021/12, Gopher (280B), SOTA LLM could do instruction-following and dialogue interaction
2022/04, Chinchilla (70B), a 4x smaller model trained on 4x more data (1.3T) outperform Gopher
2022/09, Sparrow, Building safer dialogue agents; designed to talk, answer, and search using Google, supports it with evidence

Nvidia

2019/09, Megatron-Turing NLG (530B), largest model trained with novel parallelism techniques of Nvidia

Keeping track of affordable LLMs

05/26: Falcon-40B, foundational LLM with 40 billion parameters trained on one trillion tokens, first place at huggingface Open LLM Leaderboard for now, 7B also released  (blog, model, Leaderboard)
05/25: BLIP-Diffusion, a BLIP multi-modal LLM pre-trained subject representation enables zero-shot subject-driven image generation, easily extended for novel applications (tweet, blog)
05/24: C-Eval, is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels (tweet, repo)
05/24: Guanaco-QLoRA, 33B/65B model finetuned on a single 24/48GB GPU in only 12/24h with new QLoRA 4-bit quantization, using small but with quality dataset OASST1 (tweet, repo, demo)
05/23: MMS (Massively Multilingual Speech), release by Meta AI, Can do speech2text and text speech in 1100 languages, half the word error rate of OpenAI Whisper, covers 11 times more languages. (tweet, blog, repo)
05/22: Stanford AlpacaFarm, AlpacaFarm replicates the RLHF process at a fraction of the time (<24h) and cost ($<200), enabling the research community to advance instruction following research (blog, repo)
05/22: LIMA, Less is More for Alignment (Meta AI), LLaMA 65B + 1000 standard supervised samples = {GPT4, Bard} level performance, without RLHF. (tweet, paper)
05/21: 4-bit QLoRA via bitsandbytes (4-bit base model + LoRA) (tweet)
05/20: InstructBLIP Vicuna-13B, generates text based on text and image inputs, and follows human instructions. (tweet, demo, repo)
05/18: CodeT5+, LLM for code understanding and generation (tweet, blog)
05/18: PKU-Beaver, the first chinese open-source RLHF framework developed by PKU-Alignment team at Peking University. Provide a large human-labeled dataset (up to 1M pairs) including both helpful and harmless preferences to support reproducible RLHF research. (blog, repo, data)
05/17: Tree of Thoughts(TOT), GPT-4 Reasoning is Improved 900% with this new prompting (video, paper, repo)
05/13: LaWGPT, a chinese Law LLM, extend chinese law vocab, pretrained on large corpus of law specialty (repo)
05/10: Multimodal-GPT, a multi-modal LLM Based on the open-source multi-modal model OpenFlamingo support tuning vision and language at same time, using parameter efficient tuning with LoRA (tweet, repo)
05/10: DetGPT, a multi-modal LLM addressing reasoning-based object detection problem, could interpret user instruction and automatically locate the object of interest, with only little part of whole model fine-tuned. (blog, repo)
05/10: SoftVC VITS Singing Voice Conversion, A open-source project developed to allow the developers' favorite anime characters to sing. Popular for it's been used in song generation with perticular singer's voice. (repo)
05/10: ImageBind, One Embedding Space To Bind Them All (FAIR, Meta AI), learns a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. It enables novel emergent applications ‘out-of-the-box’ with small fine-tune dataset. (tweet, blog, repo)
05/04: TIP(Dual Text-Image Prompting), a DALLE2/StableDiffusion-2 enhanced LLM that can generate coherent and authentic multimodal procedural plans toward a high-level goal (tweet)
05/04: GPTutor, a ChatGPT-powered tool for code explanation (tweet)
05/04: Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings (blog, tweet)
05/03: Modular/Mojo, a new Python-compatible language with a parallelizing compiler that can import Python libraries, combines the usability of Python with the performance of C, unlocking unparalleled programmability of AI hardware and extensibility of AI models. Only Limited notebook released for now. (tweet, blog, doc)
05/01: VPGTrans: Transfer Visual Prompt Generator across LLMs, a multi-modal LLM release by NUS for its 10 times training cose reduced. (blog, repo)
05/01: "Are Emergent Abilities of Large Language Models a Mirage?" alternative explanation for emergent abilities, strong supporting evidence that emergent abilities may not be a fundamental property of scaling AI models. (paper)
05/01: A brief history of LLaMA models (tweet, blog)
05/01: Geoffrey Hinton left Google. IBM say it can replace over 7500 current employees with AI. Chegg stock price drop 40%.
04/30: PandaLM, provide reproducible and automated comparisons between different large language models (LLMs). (tweet, repo)
04/30: Otter, a Multi-modal chatbots learn to perform tasks through rich instructions on media content (tweet, repo)
04/30: Linly-ChatFlow, Shenzhen University release Linly-ChatFlow-7B/13B/33B/65B fintune on pre-trained Chinese-LLaMA with english and chinese intruction dataset (repo)
04/29: MLC-LLM, an open framework that brings LLMs directly into a broad class of platforms (iPhone, CUDA, Vulkan, Metal) with GPU acceleration! (tweet, blog repo)
04/29: Lamini: The LLM engine for rapidly customizing models without spinning up any GPUs (tweet, blog, repo, doc)
04/29: FastChat-T5, a compact and commercial-friendly chatbot, Fine-tuned from Flan-T5, Outperforms Dolly-V2 with 4x fewer parameters (tweet, repo)
04/29: StabilityAI/StableVicuna, Carper AI from StabilityAI family release RLHF-trained version of Vicuna-13B! (tweet, blog, model)
04/29: StabilityAI/DeepFloyd IF, a powerful text-to-image model that can smartly integrate text into images, utilizes T5-XXL-1.1 as text encoder (tweet, blog)
04/29: MosaicML/SD2, Training Stable Diffusion from Scratch for <$50k with MosaicML (tweet, blog)
04/29: gpt4free, use gpt-4/3.5 free from sites (repo)
04/29: OpenRL is an open-source general reinforcement learning research framework that supports training for various tasks such as single-agent, multi-agent, and natural language. Developed based on PyTorch by chinese company 4paradigm (repo)
04/28: Chinese-LLaMA-Plus-7B, re-pretrain LLaMA on larger(120G) general corpus, fine-tune with 4M instruction dataset, bigger LoRA rank for less precision loss, beat former 13B mdoel on benchmark (repo)
04/28: AudioGPT, a multi-modal GPT model can understand audio/text/image instruction inputs and generate audio, song, style transfer speech, talking head synthesis video (blog, repo, demo)
04/28: Multimodal-GPT, released by the famous MMLab, build base on open-source multi-modal model OpenFlamingo with visual and language instructions (repo)
04/27: "Speed is all you need", generate a 512 × 512 image with 20 iterations on GPU equipped mobile devices in 12- seconds for Stable Diffusion 1.4 without INT8 quantization, 50+% latency reduced on Samsung S23 Ultra. (paper)
04/27: replit-code-v1-3b, it's a 2.7B parameters LLM trained entirely on code in 10 days, performs 40% better than comparable models on benchmark (tweet, model)
04/26: LaMini-LM, a diverse set of 15 (more coming) mini-sized models (up to 1.5B) distilled from 2.6M instructions, comparable in performance to Alpaca-7B in downstream NLP + human eval (tweet, repo, data)
04/26: huggingChat, a 30B OpenAssistant/oasst-sft-6-llama-30b-xor LLM deployed by huggingface (tweet, site, model)
04/26: LLM+P, takes in a planning problem decription, turn it into PDDL, leveraging classical planners to find a solution (tweet, paper, repo)
04/25: NeMo Guardrails, the new toolkit for easily developing trustworthy LLM-based conversational applications (tweet)
04/21: China Fudan University release its 16B LLM named MOSS-003; Moss dataset contains ~1.1M text-davinci-003 generated self-instruct dataset, include ~300k plugins dataset as text-to-image/equations/.etc, fp16 finetune on 2 A100s or 4/8-bit finetune on single 3090.  (repo)
04/21: Phoenix, a new multilingual LLM that achieves competitive performance, vast collection of popular open source dataset (repo)
04/20: UltraChat, a Informative and Diverse Multi-round Chat Data gather by THUNLP lab (repo, data)
04/20: replicate ChatGLM with efficient fine-tuning (ptunig, lora, freeze) (repo); support langchain in langchain-ChatGLM project
04/19: StableLM, 3B/7B LLM from StabilityAI (tweet, blog)
04/18: Semantic Kernel, MSFT release its contextual memory tool like langchain/gptindex (repo)
04/17: LLaVA: Large Language and Vision Assistant, Visual Instruction Tuning (blog, repo, demo)
04/17: MiniGPT-4, multi-modal LLM like GPT4, consists of a vision encoder with a pretrained ViT and Q-Former, a single linear projection layer, and an advanced Vicuna large language model (blog, repo)
04/17: TogetherCompute/RedPajama, reproduce LLaMA with 1.2 trillion tokens (blog, tweet)
04/16: LAION-AI/Open-Assistant, is an open-source chat model(includes datasets: consists of a ~161K human-annotated assistant-style conversation corpus, including 35 different languages and annotated with ~461K quality ratings) (tweet, repo, mdoels)
04/15: WebLLM, an open-source chatbot that brings LLMs like Vicuna directly onto web browsers (tweet, blog, repo)
04/12: Dolly-v2-12b, Databricks release its open source Dolly-v2-12b model, derived from EleutherAI’s Pythia-12b and fine-tuned on a ~15K record instruction corpus generated by Databricks employees, which is open source as well (blog, repo, model)
04/12: DeepSpeed Chat, DeepSpeed from MSFT support RLHF fine-tune with affordable haraware (blog)
04/12: Text-to-SQL from self-debugging explanation component (tweet)
04/11: AgentGPT, generative agents were able to simulate human-like behavior in an interactive sandbox (tweet)
04/11: AutoGPT, autonomously achieve whatever goal you set (repo)
04/11: Raven v8 14B released (tweet, model, repo)
04/09: SVDiff, diffusion fine-tune method smaller than LoRA (tweet, repo)
04/09: RPTQ, new 3 bit quantization (repo, paper)
04/08: Wonder Studio, robot beat human with kongfu (tweet)
04/08: chatGDB, chatgpt for GDB (tweet, repo)
04/08: Vicuna-7B, small yet capable (repo), Vicuna shows impressive performance against GPT4 by lastest paper of MSFTResearch (tweet)
04/07: Instruction tuning with GPT4, academic self-instruct guide from microsoft research (tweet, blog, repo, paper)
04/07: ChatPipe, Orchestrating Data Preparation Program by Optimizing Human-ChatGPT Interactions (blog)
04/07: Chinese-LLaMA-Alpaca release its 13B model (tweet)
04/07: MathPrompter, How to chatwith GPT3 Davinci API and archive better on math benmark (paper)
04/07: engshell, interact with your shell using english language (tweet)
04/07: a chinese geek fine-tune a chatglm-6b model on his wechat dialogue and blog to produce a digital version of him self (tweet)
04/06: StackLLaMA, A hands-on guide to train LLaMA with RLHF, fine-tuned on stack exchange QA data (tweet, blog, demo)
04/06: Arxiv Chat, chat with the lastest papers (tweet)
04/06: Firefly, a 1.4B/2.6B chinese chat LLM, finetune on 1.1M multi-task dataset (repo)
04/06: a chinese guide of chatgpt repo
04/06: LamaCleaner, segment anything and inpaint anything (tweet)
04/05: SAM, Meta AI release Segment Anything Model as foundation model for image segmentation, and SA-1B dataset, which is 400x larger than any existing segmentation dataset (tweet)
04/04: a beautiful cli for chatgpt (tweet)
04/04: Baize, fine-tune with LoRA using 100K dialogs ChatGPT self-chat and other opensource dataset, released 7B, 13B and 30B models (repo, tweet, demo, model)
04/03: Koala-13B, fine-tuned from LLaMA on user-shared conversations and open-source datasets, performs similarly to Vicuna (blog, demo, repo)
04/02: LMFlow, train on single 3090 for 5 hours and get your own chatgpt (blog, repo)
04/01: Alpaca-CoT, extend CoT data to Alpaca to boost its reasoning ability, provide gathered datasets (repo)
04/01: Vicuna-13B, An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality, fine-tune LLaMA on ~70K conversations from ShareGPT (blog, repo, demo, data, gptq-4-bit)
04/01: Twitter's Recommendation Algorithm (repo)
04/01: PolyglotSiri Apple Shortcut, (tweet, repo)
04/01: Official Apple Core ML Stable Diffusion Library, M-series chips beat 4090, (repo, MochiDiffusion, swift-coreml-diffusers)
03/31: BloombergGPT, 50B LLM outperform existing models on financial tasks (tweet)
03/31: HuggingGPT, as an interface for LLMs to connect AI Models for solving comlicated AI tasks (tweet, demo)
03/31: Llama-X (repo)
03/31: GPT4 UI generation (tweet)
03/30: ChatExplore (tweet)
03/30: ColossalChat, from ColossalAI (demo, tweet, medium, repo, serve)
03/30: ChatGLM-6B, from THUDM(Tsinghua University), code and data not release (repo, model)
03/29: Uncle Rabbit, the first conversational holographic AI (tweet, blog)
03/29: chatgpt instead of siri (tweet)
03/29: LLaMA-Adapter, fine-tuning LLaMA with 1.2M learnable parameters in 1 hour on 8 A100 (tweet, repo, demo)
03/28: Chinese-LLaMA-Alpaca, add 20K chinese sentencepiece tokens to vocab and pre-trained LLaMA in 2 steps, fine-tuned LLaMA on a 2M chinese corpus using Alpaca-LoRA, 7B model released, dataset not (repo, tweet, blog, model)
03/28: gpt4all, fine-tune LLaMa using LoRA with ~800k gpt3.5-turbo generations, include clean assistant data including code, stories and dialogue (repo, model, data)
03/24: Dolly, Databricks fine-tune alpaca dataset on gpt-j-6b (repo)
03/22: Alpaca-LoRA-Serve, gradio based chatbot service (tweet, repo)
03/22: Alpaca-LoRA, reproducing the Stanford Alpaca results using low-rank adaptation(LoRA) on RTX4090 and run on a Raspberry Pi 4 (tweet, repo, demo, model, blog, reproduce tweet, zhihu, sina, explain)
03/22: BELLE, fine-tune BLOOMZ-7B1-mt and LLaMA(7B/13B) on a 1.5M chinese dataset generate in a alpaca way, (repo, model)
03/17: instruct-gpt-j, NLPCloud team fine-tune GPT-J using Alpaca's dataset (blog, model)
03/13: Stanford Alpaca, fine-tune LLaMA 7B with a 52K single-turn instruction-followling dataset generate from OpenAI’s text-davinci-003  (blog, repo)
03/11: ChatIE, solving Zero-Shot Information Extraction problem by enhancing ChatGPT with CoT prompting, achieve good performance on primary IE benchmarks (repo)
prompt engineering guide (blog), openai best practices (blog), prompt prefect (blog), prompt searching (repo), PromptInject (repo), auto prompt engineering (blog)
peft: State-of-the-art Parameter-Efficient Fine-Tuning (PEFT) methods (repo)
GPTQ-for-LLaMa: 4 bits quantization of LLaMA using GPTQ (repo)
llama.cpp: Inference of LLaMA model in pure C/C++, support different hardware platform & models, support 4-bit quantization using ggml format (repo, alpaca.cpp); support python bindings (llama-cpp-python, pyllamacpp, llamacpp-python )
llama_index: connect LLM with external data (repo), like langchain (repo)
llama-dl: high speed download of LLaMA model (repo(deprecated), model)
text-generation-webui: A gradio web UI for deploy LLMs like GPT-J, LLaMA (repo)
tldream/lama-cleaner: tiny little diffusion drawing app (repo1, repo2)
A1111-Web-UI-Installer: A gradio web UI for deploy stable diffusion models (repo)

Instruction and conversational datasets

https://github.com/allenai/natural-instructions: AllenAI NI dataset, a collection of over 1.6K tasks and their natural language definitions/instructions, more
https://huggingface.co/datasets/bigscience/P3: BigScience P3 (Public Pool of Prompts), a collection of prompted English datasets covering 55 tasks.
https://laion.ai/blog/oig-dataset/: The Open Instruction Generalist (OIG-43M) dataset is a large open source instruction dataset that currently contains ~43M instructions.
https://huggingface.co/datasets/databricks/databricks-dolly-15k: Dolly v2.0 dataset by DataBricks.
https://huggingface.co/datasets/OpenAssistant/oasst1: OpenAssistant Conversations (OASST1), a human-generated, human-annotated assistant-style conversation corpus consisting of 161k messages in 35 different languages, annotated with 461k quality ratings, resulting in over 10,000 fully annotated conversation trees.
https://huggingface.co/datasets/Anthropic/hh-rlhf: Anthropic Helpful and Harmless (HH-RLHF) datasets, include (1) Human preference data about helpfulness and harmlessness, (2) Human-generated and annotated red teaming dialogues
https://huggingface.co/datasets/jeffwan/sharegpt_vicuna
https://huggingface.co/datasets/Hello-SimpleAI/HC3: Human ChatGPT Comparison Corpus (HC3)
https://huggingface.co/datasets/tatsu-lab/alpaca: 50k instruction following dataset from ChatGPT
https://huggingface.co/datasets/WizardLM/evol_instruct_70k
https://github.com/google-research/FLAN/tree/main/flan/v2: The Flan Collection dataset

Pre-training datasets

the Pile
RedPajama-Data-1T
C4
mC4
the Stack

Training Library

CarperAI/trlx: a distributed training framework language models with RLHF
Microsoft/DeepSpeedChat
ColossalAI/ColossalChat
PyTorch FSDP
ALiBi: [Train Short, Test Long: Attention with Linear Biases (ALiBi) Enables Input Length Extrapolation], handle extremely long inputs
NVIDIA/FasterTransformer: highly optimized transformer-based encoder and decoder component
MosaicML: Composer, llm-foundry

Evaluation Benchmark

EleutherAI/lm-evaluation-harness: industry-standard ICL evaluation codebase, mpt example.
Stanford HELM
Google/BIG-bench
BIG-Bench-Hard
MMLU
google-research-datasets/seahorse: Seahorse is a compound dataset for summarization evaluation. It consists of 96K summaries with human ratings along 6 quality dimensions.

Evaluation Benchmark

EleutherAI/lm-evaluation-harness: industry-standard ICL evaluation codebase, mpt example.
Stanford HELM
Google/BIG-bench
BIG-Bench-Hard
MMLU
google-research-datasets/seahorse: Seahorse is a compound dataset for summarization evaluation. It consists of 96K summaries with human ratings along 6 quality dimensions.

LLM Evaluation Reference
Traditionally, language model performance is measured by measuring perplexity, cross entropy, and bits-per-character (BPC). Large Language Models, on the other hand, have been shown to outperform these benchmarks and unlock new abilities such as arithmetic, few-shot learning, and multi-step reasoning. Nevertheless, these LLMs are not without flaws, exhibiting biases and producing plausible misinformation.
Recent benchmarks address these issues by testing LLMs for logical and common sense reasoning, dataset-specific bias, the ability of models to keep track of information, and downstream tasks without task-specific gradient optimization. A few examples of such benchmarks are CoQA, LAMBDA, HELLASWAG, LogiQA. These benchmarks provide methods for evaluating LLMs for mismatches between the behavior we want LLMs to exhibit and the behavior we observe in practice as a result of the training objectives and data we use.

https://instruction-tuning-with-gpt-4.github.io/: ask GPT-4 to rate LLM response from 1 to 10, User-Oriented-Instructions-252, Vicuna-Instructions-80; Can Large Language Models Be an Alternative to Human Evaluation? paper shows ChatGPT rating human-write stories similar to human english teachers, test on 400 stories;
https://github.com/WeOpenML/PandaLM/: finetune alpaca-52K dataset on several 7B model, using ChatGPT generate 300K comparison score for each 2 response pair and build reward model base on it
https://lmsys.org/blog/2023-05-10-leaderboard/: Chatbot Arena leaderboard, Elo ratings of all 13 models, which are based on the 13K anonymous voting data; network constraint in china; [Directly Comparison with community votes !]
https://github.com/ymcui/Chinese-LLaMA-Alpaca/blob/pro/examples/README.md: a chinese benchmark contains 10 tasks (20 samples for each, sum as 200) [Good start]
https://github.com/CLUEbenchmark/SuperCLUE: a chinese benchmark with 10 basic ability, 10 chinese ability, 50+ specialty, but Not providing any evaluation detail or dataset or dataset size [Unreasonable project, not recommended]
https://openai.com/research/truthfulqa: TruthfulQA, a OpenAI benchmark for Measuring how models mimic human falsehoods, which comprises 817 questions that span 38 categories, including health, law, finance and politics. [Special perspective!]
https://github.com/EleutherAI/lm-evaluation-harness: 200+ task implemented(hundreds/thousands/ten-thousands for each), support transformers, NeoX, .etc, more detail see this report Evaluating-Large-Language-Models-LLMs-with-Eleuther-AI [Cool! Great Job]
https://github.com/google/BIG-bench: collaborative benchmark contains more than 200 tasks [Sound Great]
https://github.com/Hello-SimpleAI/chatgpt-comparison-detection: tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3).
https://github.com/hendrycks/test: Multi-task Language Understanding(MMLU), Sep 2020, a multitask accuracy test covers 57 tasks including elementary mathematics, US history, computer science, law, and more, leaderboard, [good open source but old]
https://github.com/Felixgithub2017/MMCU: Multi-task Chinese Understanding(MMCU), Apr 2023, measure the multitask accuracy of Chinese LLMs,  encompasses four major domains, including medicine, law, psychology, and education, with 15 subtasks in medicine and 8 subtasks in education. Email to author for apply free dataset download. [Seems reasonable]
https://github.com/microsoft/AGIEval: MSFT release a benchmark derived from 20 official, public, hign-standard qualification human exams, like gaokao, SAT. AGIEval v1.0 contains 20 tasks, including two cloze tasks (Gaokao-Math-Cloze and MATH) and 18 multi-choice question answering tasks (the rest).
https://gpt4all.io/index.html: open source project gpt4all benchmark: BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA
https://github.com/bigcode-project/bigcode-evaluation-harness: OpenAI's code-cushman-001, (tweet)
https://openai.com/research/evaluating-large-language-models-trained-on-code: OpenAI Codex performance on HumanEval.
https://github.com/rlancemartin/auto-evaluator: Evaluation tool for LLM QA chains
https://github.com/Hannibal046/Awesome-LLM/blob/main/paper_list/evaluation.md: a evaluation list, contains traditional NLP tasks
[GPT-4 Technical Report, OpenAI, 2023.03]
[Sparks of Artificial General Intelligence: Early experiments with GPT-4, MSFT, 2023.04]
[PaLM 2 Technical Report, Google, 2023.05]

Evaluation Dilemma

Benchmark dataset may have been collected in pre-training/post-training, which make it can't evaluate LLM properly
LLM could involve with specific data, evaluation only take place on one snapshot
Too many benchmark to evaluate if you want a comprehensive evaluation
Generative LLM alway need human evaluation or prompting actions, which cost too much

Evaluation In a Afforable Way
Human Exams

professional language proficiency exams (PaLM2)

Chinese: HSK汉语水平考试
Japanese: J-Test日本语检定
Frech: TCF Test...
Spanish: DELE C2...
German: Goethe-Zertifikat C2
Italian: PLIDA C2...

professional exams (GPT-4)

SAT
GRE
Medical Knowledge Self-Assesment Program
AP Art/Biology/Calculus/Chemistry/English Literature/Environment/Physics/Psychology/Statistics/History/Government...
...
Leetcode(Easy/Midium/Hard)

https://github.com/microsoft/AGIEval: MSFT release a benchmark derived from 20 official, public, hign-standard qualification human exams, like gaokao, SAT. AGIEval v1.0 contains 20 tasks, including two cloze tasks (Gaokao-Math-Cloze and MATH) and 18 multi-choice question answering tasks (the rest).

Recommandation: Chinese HSK, Gaokao, SAT (https://github.com/microsoft/AGIEval)
Purpose: test for basic language and knowledge understanding
Evaluation Method: zero-shot multiple choice questions(Prompt+Auto), free-text response(Human)
Evalution Metrics: Accuracy/Scores of single/multiple choice questions

Question Answering and Classification

English QA and classification tasks(one-shot setting)

Open-domain closed-book question answering tasks: TriviaQA (Joshi et al., 2017), Natural Questions2
(Kwiatkowski et al., 2019), and WebQuestions (Berant et al., 2013)
Cloze and completion tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), and StoryCloze
(Mostafazadeh et al., 2016)
Winograd-style tasks: Winograd (Levesque et al., 2012) and WinoGrande (Sakaguchi et al., 2021)
Reading comprehension: SQuAD v2 (Rajpurkar et al., 2018) and RACE (Lai et al., 2017)
Common sense reasoning: PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), and OpenBookQA (Mihaylov
et al., 2018)
SuperGLUE (Wang et al., 2019)
Natural language inference: Adversarial NLI (ANLI; Nie et al., 2020)

Multilingual QA (one-shot and no-content setting): TyDi QA (Clark et al., 2020)
Multilingual toxicity classification

Toxicity classification with CivilComments
Multilingual toxicity classification with Jigsaw Multilingual

https://openai.com/research/truthfulqa: TruthfulQA, a OpenAI benchmark for Measuring how models mimic human falsehoods(based on misconceptions and biases they may have), which comprises 817 questions that span 38 categories(with a median of 7 questions and a mean of 21.5 questions per category), including health, law, finance, science and politics.

https://yonatanbisk.com/piqa/data/: 20,000 QA pairs that are either multiple-choice or true/false questions, main works on daily physical interaction and common sense reasoning

Recommandation: TruthfulQA (https://github.com/sylinrl/TruthfulQA) | PIQA (https://leaderboard.allenai.org/physicaliqa/submissions/get-started)
Purpose: QA is about the common test bed, TruthfulQA do falsehood/hallucinations evaluation, PIQA do common sense evaluation and frequently used;
Evaluation Method: BLEURT, GPT-Judge/Human | One-Shot Prompting, Accuracy(Binary Choice)

Reasoning(Common Sense, Math)

representative reasoning datasets in a few-shot setting: WinoGrande (Sakaguchi et al., 2021), ARC-C (Clark et al., 2018), DROP (Dua et al.,2019), StrategyQA (Geva et al., 2021), CommonsenseQA (CSQA; Talmor et al., 2019), XCOPA (Ponti et al., 2020), and BIG-Bench (BB) Hard (Suzgun et al., 2022).

Multilingual common sense reasoning: XCOPA
BIG-Bench (BB) Hard: 23 tasks from 200+, where LLM performed below average human, like multi-step arithmetic problems(multistep_arithmetic)

Mathematical reasoning

MATH (Hendrycks et al., 2021), which contains 12,500 problems from high school competitions in 7 mathematics subject areas
GSM8K (Cobbe et al., 2021), a dataset of 8,500 grade school math word problems
MGSM (Shi et al., 2023), a multilingual version of GSM8K with translations of a subset of examples into ten typologically diverse languages.

https://github.com/suzgunmirac/BIG-Bench-Hard: 23 challenging BIG-Bench tasks from 200+ BIG-Bench (https://github.com/google/BIG-bench);

Recommandation: BIG-Bench-Hard (https://github.com/suzgunmirac/BIG-Bench-Hard, https://github.com/google/BIG-bench/blob/main/bigbench/evaluate_task.py)
Purpose: Evaluate ability of LLM to reason, to combine multiple pieces of information, and to make logical inferences | From Paper2(Early Experiments) analysis, even GPT-4 can do only simple math right now, much arithmetic and calculation mistakes on MATH;
Evaluation Method: chain-of-thought (CoT)

Evalution Metrics: Accuracy/Score, Evaluate the truth value of a random Boolean expression consisting of Boolean constants (True, False) and basic Boolean operators (and, or, not).

Coding

Code Generation: 3 coding datasets: HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and ARCADE (Yin et al., 2022),
Multilingual Evaluation: BabelCode (Orlanski et al., 2023) which translates HumanEval into a variety of other programming languages including c++, java, go, haskell and julia.
Leetcode questions(100 for each level)
https://github.com/openai/human-eval: HumanEval, a docstring-to-code dataset consisting of 164 coding problems that test various aspects of programming logic and proficiency

Recommandation: HumanEval (https://github.com/openai/human-eval), most widely used, Python code;
Purpose: Code language models are among the most economically significant and widely-deployed LLMs today
Evaluation Method: Excution in a robust security sandbox (https://github.com/openai/human-eval, https://github.com/bigcode-project/bigcode-evaluation-harness)
Evalution Metrics: Pass@1, Pass@K

Translation/Multi-lingual

WMT21 Experimental Setup: automatic metric using BLEURT, human metric using Multidimensional Quality Metrics (MQM) with hired professional translators
Recommandation:
Purpose:
Evaluation Method:
Evalution Metrics:

Natural language generation

https://github.com/csebuetnlp/xl-sum: XLSum (Hasan et al., 2021), which asks a model to summarize a news article for 44 languages
WikiLingua (Ladhak et al., 2020), which focuses on generating section headers for step-by-step instructions from WikiHow
XSum (Narayan et al., 2018), which tasks a model with generating a news article’s first sentence
https://github.com/ymcui/Chinese-LLaMA-Alpaca/blob/pro/examples/README.md: Chinese-LLaMA-Alpaca released a 200 sample chinese sample for 20 generation task;
Recommandation: XLSum (https://github.com/csebuetnlp/xl-sum) | Chinese-LLaMA-Alpaca-200 (https://github.com/ymcui/Chinese-LLaMA-Alpaca/blob/pro/examples/README.md)
Purpose: Evaluate models’ generation quality, naturalness and smoothness;
Evaluation Method: 1-shot prompting | zero-shot prompting
Evalution Metrics: ROUGE-L; SentencePiece-ROUGE-2, an extension of ROUGE that handles non-Latin characters using a SentencePiece tokenizer | Human Elo rating from 1 to 10

测试任务
详细样例
样例数
中文Alpaca-7B
中文Alpaca-13B
中文Alpaca-Plus-7B

💯总平均分
-
200
65.1
70.6
👍🏻75.3

知识问答
QA.md
20
66
74
👍🏻80

开放式问答
OQA.md
20
👍🏻79
74
👍🏻78

数值计算、推理
REASONING.md
20
31
👍🏻50
45

诗词、文学、哲学
LITERATURE.md
20
68
73
👍🏻76

音乐、体育、娱乐
ENTERTAINMENT.md
20
68
74
👍🏻79

写信、写文章
GENERATION.md
20
76
👍🏻81
👍🏻81

文本翻译
TRANSLATION.md
20
76
78
👍🏻82

多轮交互
DIALOGUE.md
20
👍🏻83
73
👍🏻84

代码编程
CODE.md
20
57
👍🏻64
59

伦理、拒答
ETHICS.md
20
47
65
👍🏻89

LLM Arena (side by side comparison)

Purpose: Arena perform direct comparison while point/scores compare LLM in a indirect way, which gives a clearer preference;
Evaluation Method: Elo rating by online voters
https://lmsys.org/blog/2023-05-10-leaderboard/

<div class="header">
<h1>Large Models for All</h1>
<h2>We're building something big...</h2>
<h2>Stay tuned! </h2>
<div>

<iframe style="width:100%;height:auto;min-width:600px;min-height:400px;" src="https://star-history.com/embed?secret=#shm007g/LLaMA-Cult-and-More&Date" frameBorder="0"></iframe>

<thanksto> https://github.com/mli </thanksto>

<!DOCTYPE html>
<style>
    body {
      background-image: url('https://boson.ai/bg.jpg');
      background-attachment: fixed;
      background-size: cover;
    }

    .header {
        padding-top: 50px;
        margin: auto;
        width: 60%;
        font-family: Arial,Helvetica,sans-serif;
    }

    h1 {
        font-size: 110px;
        color: #fff;
    }
    h2 {
        font-size: 80px;
        color: #aaa;
    }

    </style>

<html>
<body>

    <div class="header">
    <h1>Large Models for All</h1>
    <h2>We're building something big...</h2>
    <h2>Stay tuned! </h2>
    <div>

    <picture>
        <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=shm007g/LLaMA-Cult-and-More&type=Date&theme=dark" />
        <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=shm007g/LLaMA-Cult-and-More&type=Date" />
        <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=shm007g/LLaMA-Cult-and-More&type=Date" />

    </picture>

    <input type="hidden" id="custId" name="custId" value="thanks to https://github.com/boson-ai">
    <h2></h2>

</body>
</html>


"might" or "might not"?

Hello!
I have a doubt about a phrase in this paragraph:
"If the client code runs in a web browser, the browser might need to support cross-origin resource sharing (CORS) to enable code that executes within the web browser to access data in a different domain from the one that served the page. Some older browsers and some data stores don't support CORS, and code that runs in these browsers might be able to use a valet key to provide access to data in a different domain, such as a cloud storage account."
By semantic it should be "might not be able to", am I right?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d94a95d1-5371-4c6a-87d6-00fae7dfb401
Version Independent ID: d9abac19-dfd7-18d7-2479-42fe5476b569
Content: Valet Key pattern - Cloud Design Patterns
Content Source: docs/patterns/valet-key.md
Service: architecture-center
Sub-service: cloud-fundamentals
GitHub Login: @dragon119
Microsoft Alias: pnp

@Kamil-Zakiev Thanks for the feedback! We are currently investigating and will update you shortly.

@Kamil-Zakiev Thanks for catching that! We have worked up a PR to address this. Once merged, the change will reflect in a couple of hours.


ci: debug e2e after

:tada: This PR is included in version 1.75.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Bug in quickwit search stream StorageDirectory only supports async reads

Copy pasted from https://github.com/quickwit-oss/quickwit/discussions/1357#discussioncomment-2687107
am able to ingest data in quickwit and search . However when I search using curl command , I am getting read async error.
What could go wrong here.
heena@Clickhouse1:~/quickwit-v0.2.1$ ./quickwit index search --index hackernews_5 --query Ambulance 2022-05-04T13:25:27.169Z ERROR quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads" 2022-05-04T13:25:27.171Z ERROR quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads" {   "numHits": 1,   "hits": [     {       "by": [         "sgk284"       ],       "id": [         2923885       ],       "kids": [         2923989,         2925247,         2924320,         2925442,         2924224,         2923994,         2924209,         2924702,         2925235,         2925010,         2924319,         2924638,         2925781,         2923943,         2924298       ],       "score": [         622       ],       "text": [         ""       ],       "time": [<PHONE_NUMBER>       ],       "title": [         "Icon Ambulance"       ],       "type": [         "story"       ],       "url": [         "https://plus.google.com/107117483540235115863/posts/gcSStkKxXTw"       ]     }   ],   "elapsedTimeMicros": 77324,   "errors": [     "SplitSearchError { error: \"Internal error: An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: \"ccf34dbac4614904b1124b751756dab8.term\"'.\", split_id: \"01G26NHMCV1BAP61AS006H7A75\", retryable_error: true }",     "SplitSearchError { error: \"Internal error: An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: \"6dc68fd1122c44a985ccf5348907c5f8.term\"'.\", split_id: \"01G26NK8YX0DM4YSVH6J9YD1GN\", retryable_error: true }"   ] }
The output with curl command to search the same keyword.
heena@Clickhouse1:~/quickwit-v0.2.1$ curl "http://<IP_ADDRESS>:7280/api/v1/hackernews_5/search/stream?query=Ambulance&outputFormat=csv&fastField=id" curl: (18) transfer closed with outstanding read data remaining heena@Clickhouse1:~/quickwit-v0.2.1$
Attached the console logs when queried the commands ,This might be helpful
2022-05-04T13:24:03.927Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:13.927Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "google", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: ClickHouseRowBinary, partition_by_field: None }
2022-05-04T13:24:13.927Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:13.968Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:13.969Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:13.970Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:13.972Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:14.006Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:14.006Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:14.007Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: `Internal error: `An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "ccf34dbac4614904b1124b751756dab8.term"'`.`.
2022-05-04T13:24:14.009Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:49.399Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "google.com", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: Csv, partition_by_field: None }
2022-05-04T13:24:49.400Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:49.442Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.442Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.443Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:49.452Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:49.494Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.495Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.496Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: `Internal error: `An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "ccf34dbac4614904b1124b751756dab8.term"'`.`.
2022-05-04T13:24:49.503Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:26:29.659Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "Ambulance", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: Csv, partition_by_field: None }
2022-05-04T13:26:29.661Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:26:29.705Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.706Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.707Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:26:29.713Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:26:29.756Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.757Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.757Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: `Internal error: `An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "ccf34dbac4614904b1124b751756dab8.term"'`.`.
2022-05-04T13:26:29.761Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed

@HeenaBansal2009 Can you add the first line of stdout? It contains the sha1 of the version you used? Is it main or is the released 0.2.1?
Also did you use S3 or the local file storage?

@HeenaBansal2009 Can you add the first line of stdout? It contains the sha1 of the version you used? Is it main or is the released 0.2.1?
Also did you use S3 or the local file storage?

@fulmicoton , I used the local file storage. I am using 0.2.1
heena@Clickhouse1:~/quickwit-v0.2.1$ ./quickwit --version Quickwit 0.2.1 (commit-hash: a857636) heena@Clickhouse1:~/quickwit-v0.2.1$
Pasting the logs here .
``heena@Clickhouse1:~/quickwit-v0.2.1$ ./quickwit service run searcher
2022-05-04T20:07:19.736Z  INFO quickwit: version="0.2.1" commit="a857636"
2022-05-04T20:07:19.747Z  WARN quickwit_config::config: Seed list is empty.
2022-05-04T20:07:19.747Z  INFO quickwit_cli: Loaded Quickwit config. config_uri=file:///home/heena/quickwit-v0.2.1/config/quickwit.yaml config=QuickwitConfig { version: 0, node_id: "node-patient-lLKH", listen_address: "<IP_ADDRESS>", rest_listen_port: 7280, peer_seeds: [], data_dir_path: "./qwdata", metastore_uri: "file:///home/heena/quickwit-v0.2.1/qwdata/indexes", default_index_root_uri: "file:///home/heena/quickwit-v0.2.1/qwdata/indexes", indexer_config: IndexerConfig { split_store_max_num_bytes: Byte(100000000000), split_store_max_num_splits: 1000 }, searcher_config: SearcherConfig { fast_field_cache_capacity: Byte(1000000000), split_footer_cache_capacity: Byte(500000000), max_num_concurrent_split_streams: 100 }, storage_config: None }

Connectivity checklist
✔ metastore
2022-05-04T20:07:19.755Z  INFO quickwit_cluster::cluster: Create new cluster. node_id="node-patient-lLKH" listen_addr=<IP_ADDRESS>:7280
2022-05-04T20:07:19.761Z  INFO quickwit_serve: Searcher ready to accept requests at http://<IP_ADDRESS>:7280/
2022-05-04T20:07:19.762Z  INFO quickwit_serve::rest: Starting REST service. rest_addr=<IP_ADDRESS>:7280
2022-05-04T20:07:19.763Z  INFO quickwit_serve::grpc: Start gRPC service. grpc_addr=<IP_ADDRESS>:7281
2022-05-04T20:07:24.970Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "Ambulance", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: Csv, partition_by_field: None }
2022-05-04T20:07:24.984Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T20:07:24.991Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:07:24.997Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:07:25.005Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T20:07:25.006Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:07:25.006Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:07:25.007Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: Internal error: An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "6dc68fd1122c44a985ccf5348907c5f8.term"'..
2022-05-04T20:07:25.107Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T20:07:25.117Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T20:08:20.383Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "Ambulance", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: Csv, partition_by_field: None }
2022-05-04T20:08:20.384Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T20:08:20.433Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:08:20.433Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:08:20.434Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start:<PHONE_NUMBER>, split_footer_end:<PHONE_NUMBER> }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T20:08:20.448Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T20:08:20.476Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:08:20.476Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T20:08:20.477Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: Internal error: An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "ccf34dbac4614904b1124b751756dab8.term"'..
2022-05-04T20:08:20.481Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed

thank you this is super helpful!

@HeenaBansal2009 this is quite mysterious. Can you provide even more logs? What happened before the error...
Don't worry about being verbose. You can attach a file if necessary.
RUST_LOG=debug might help too.

Flushing the status of my investigation.
I have not been able to reproduce yet. The logs suggests everything is working somewhat correctly, but the warmup phase did not work properly for two splits.

@HeenaBansal2009
Also can you share the content of the meta.json that you obtain after running:
./quickwit split extract --index hackernews_5 --split 01G26NK8YX0DM4YSVH6J9YD1GN --target-dir .
as well as the current index configuration.
If you don't have your config file, you can send the qwdata/indexes/hackernews_5/metastore.json file.
My current suspicion is a bug triggered by a change of schema.

meta.json
{   "index_settings": {     "docstore_compression": "lz4"   },   "segments": [     {       "segment_id": "6dc68fd1-122c-44a9-85cc-f5348907c5f8",       "max_doc": 421980,       "deletes": null     }   ],   "schema": [     {       "name": "id",       "type": "u64",       "options": {         "indexed": true,         "fieldnorms": false,         "fast": "single",         "stored": true       }     },     {       "name": "type",       "type": "text",       "options": {         "indexing": {           "record": "basic",           "fieldnorms": true,           "tokenizer": "raw"         },         "stored": true       }     },     {       "name": "by",       "type": "text",       "options": {         "indexing": {           "record": "position",           "fieldnorms": true,           "tokenizer": "default"         },         "stored": true       }     },     {       "name": "time",       "type": "i64",       "options": {         "indexed": true,         "fieldnorms": false,         "fast": "single",         "stored": true       }     },     {       "name": "text",       "type": "text",       "options": {         "indexing": {           "record": "position",           "fieldnorms": true,           "tokenizer": "raw"         },         "stored": true       }     },     {       "name": "kids",       "type": "i64",       "options": {         "indexed": true,         "fieldnorms": false,         "fast": "multi",         "stored": true       }     },     {       "name": "url",       "type": "text",       "options": {         "indexing": {           "record": "position",           "fieldnorms": true,           "tokenizer": "default"         },         "stored": true       }     },     {       "name": "score",       "type": "i64",       "options": {         "indexed": true,         "fieldnorms": false,         "fast": "single",         "stored": true       }     },     {       "name": "title",       "type": "text",       "options": {         "indexing": {           "record": "position",           "fieldnorms": true,           "tokenizer": "default"         },         "stored": true       }     }   ],   "opstamp": 421981 }
metastore.json
`
{
"version": "0",
"index": {
"version": "1",
"index_id": "hackernews_5",
"index_uri": "file:///home/heena/quickwit-v0.2.1/qwdata/indexes/hackernews_5",
"checkpoint": {
".cli-ingest-source": {}
},
"doc_mapping": {
"field_mappings": [
{
"name": "id",
"type": "u64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "type",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "basic"
},
{
"name": "by",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "time",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "text",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "position"
},
{
"name": "kids",
"type": "array",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "url",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "score",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "title",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
}
],
"tag_fields": [],
"store_source": false
},
"indexing_settings": {
"timestamp_field": "time",
"commit_timeout_secs": 60,
"split_num_docs_target": 10000000,
"merge_enabled": true,
"merge_policy": {
"demux_factor": 8,
"merge_factor": 10,
"max_merge_factor": 12
},
"resources": {
"num_threads": 1,
"heap_size":<PHONE_NUMBER>
}
},
"search_settings": {
"default_search_fields": [
"title",
"by",
"text",
"url"
]
},
"create_timestamp":<PHONE_NUMBER>,
"update_timestamp":<PHONE_NUMBER>
},
"splits": [
{
"split_state": "Published",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NHEB10T2DX37288EKX0SJ",
"num_docs": 327347,
"size_in_bytes": 432879779,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270323695,
"end": 278910648
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N674ASKNNVG8DQKDYKRJM",
"num_docs": 387410,
"size_in_bytes": 502431085,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 312636135,
"end": 322041208
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26MQHW2BPQGPJJK3V44TJVJ",
"num_docs": 568590,
"size_in_bytes": 719620809,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 438230039,
"end": 438328080
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26MYWBR1YCXASA89KZ8F8GW",
"num_docs": 411217,
"size_in_bytes": 519948004,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 317238052,
"end": 326780516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N81RVV4BNM0C2MWER0Z78",
"num_docs": 478959,
"size_in_bytes": 625017918,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 388813589,
"end": 388900337
}
},
{
"split_state": "Published",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NHMCV1BAP61AS006H7A75",
"num_docs": 3678305,
"size_in_bytes":<PHONE_NUMBER>,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N0Q0X5PDV52YA6EKKBGFF",
"num_docs": 271265,
"size_in_bytes": 344296679,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 212666409,
"end": 218978775
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N2HPB28PTYA8TDTKRP8D6",
"num_docs": 279867,
"size_in_bytes": 361323563,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 225477171,
"end": 232727176
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N4CBC75E4TY1HNZMS7J9C",
"num_docs": 396428,
"size_in_bytes": 511474431,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 321116808,
"end": 331051249
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N9WDRQMEM9FQ68ZH04Y4G",
"num_docs": 307345,
"size_in_bytes": 400861275,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 250380683,
"end": 257950016
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NDS1DD9ZRN5GQB1F8HKPK",
"num_docs": 331782,
"size_in_bytes": 429910307,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270652419,
"end": 278863516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NFKNBXHCCJXPZSTNHBBY8",
"num_docs": 245442,
"size_in_bytes": 321793812,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 200656781,
"end": 206628263
}
},
{
"split_state": "Published",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NK8YX0DM4YSVH6J9YD1GN",
"num_docs": 421980,
"size_in_bytes": 559498837,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 349970236,
"end": 350048435
}
}
]
}
{
"version": "0",
"index": {
"version": "1",
"index_id": "hackernews_5",
"index_uri": "file:///home/heena/quickwit-v0.2.1/qwdata/indexes/hackernews_5",
"checkpoint": {
".cli-ingest-source": {}
},
"doc_mapping": {
"field_mappings": [
{
"name": "id",
"type": "u64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "type",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "basic"
},
{
"name": "by",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "time",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "text",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "position"
},
{
"name": "kids",
"type": "array",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "url",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "score",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "title",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
}
],
"tag_fields": [],
"store_source": false
},
"indexing_settings": {
"timestamp_field": "time",
"commit_timeout_secs": 60,
"split_num_docs_target": 10000000,
"merge_enabled": true,
"merge_policy": {
"demux_factor": 8,
"merge_factor": 10,
"max_merge_factor": 12
},
"resources": {
"num_threads": 1,
"heap_size":<PHONE_NUMBER>
}
},
"search_settings": {
"default_search_fields": [
"title",
"by",
"text",
"url"
]
},
"create_timestamp":<PHONE_NUMBER>,
"update_timestamp":<PHONE_NUMBER>
},
"splits": [
{
"split_state": "Published",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NHEB10T2DX37288EKX0SJ",
"num_docs": 327347,
"size_in_bytes": 432879779,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270323695,
"end": 278910648
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N674ASKNNVG8DQKDYKRJM",
"num_docs": 387410,
"size_in_bytes": 502431085,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 312636135,
"end": 322041208
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26MQHW2BPQGPJJK3V44TJVJ",
"num_docs": 568590,
"size_in_bytes": 719620809,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 438230039,
"end": 438328080
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26MYWBR1YCXASA89KZ8F8GW",
"num_docs": 411217,
"size_in_bytes": 519948004,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 317238052,
"end": 326780516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N81RVV4BNM0C2MWER0Z78",
"num_docs": 478959,
"size_in_bytes": 625017918,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 388813589,
"end": 388900337
}
},
{
"split_state": "Published",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NHMCV1BAP61AS006H7A75",
"num_docs": 3678305,
"size_in_bytes":<PHONE_NUMBER>,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N0Q0X5PDV52YA6EKKBGFF",
"num_docs": 271265,
"size_in_bytes": 344296679,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 212666409,
"end": 218978775
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N2HPB28PTYA8TDTKRP8D6",
"num_docs": 279867,
"size_in_bytes": 361323563,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 225477171,
"end": 232727176
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N4CBC75E4TY1HNZMS7J9C",
"num_docs": 396428,
"size_in_bytes": 511474431,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 321116808,
"end": 331051249
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26N9WDRQMEM9FQ68ZH04Y4G",
"num_docs": 307345,
"size_in_bytes": 400861275,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 250380683,
"end": 257950016
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NDS1DD9ZRN5GQB1F8HKPK",
"num_docs": 331782,
"size_in_bytes": 429910307,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270652419,
"end": 278863516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NFKNBXHCCJXPZSTNHBBY8",
"num_docs": 245442,
"size_in_bytes": 321793812,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 200656781,
"end": 206628263
}
},
{
"split_state": "Published",
"update_timestamp":<PHONE_NUMBER>,
"version": "1",
"split_id": "01G26NK8YX0DM4YSVH6J9YD1GN",
"num_docs": 421980,
"size_in_bytes": 559498837,
"time_range": {
"start":<PHONE_NUMBER>,
"end":<PHONE_NUMBER>
},
"create_timestamp":<PHONE_NUMBER>,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 349970236,
"end": 350048435
}
}
]
}
`

@fmassot , I am unable to attach json files here , pasted the content.
This is my index configuration file:
#
# Index config file for gh-archive dataset.
#
version: 0

index_id:  hackernews_5

doc_mapping:
  store_source: false
  field_mappings:
    - name: id
      type: u64
      fast: true
    - name: type
      type: text
      tokenizer: raw
    - name: by
      type: text
      tokenizer: default
      record: position
    - name: time
      type: i64
      stored: true
      indexed: true
      fast: true
    - name: text
      type: text
      tokenizer: raw
      record: position
    - name: kids
      type: array<i64>
    - name: url
      type: text
      tokenizer: default
      record: position
    - name: score
      type: i64
      fast: true
    - name: title
      type: text
      tokenizer: default
      record: position

indexing_settings:
  timestamp_field: time

search_settings:
  default_search_fields: [title,by,text,url ]

RUST_LOG=debug

Where I can provide this level of logs . I don't see  anything with help option. Just wandering if I ma putting it at right place or do you want me to add somewhere else .
/quickwit service run searcher RUST_LOG=debug
error: Found argument 'RUST_LOG=debug' which wasn't expected, or isn't valid in this context

USAGE:
    quickwit service run searcher [OPTIONS] --config <CONFIG>

For more information try --help

Ah sorry. This is controlled by an environment variable.
If you are on linux or mac os, the following should work.
export RUST_LOG=debug

Ah sorry. This is controlled by an environment variable. If you are on linux or mac os, the following should work.
export RUST_LOG=debug

Please see attached.
Quickwit extended logs.docx

I did not find a change in schema. This might be a bug in tantivy's sstable based dictionary. I see that you set the raw tokenizer on the text field which I assume can be large.
There might a strange edge case hitting on super large tokens that breaks our sstable dictionary in some way.
I have enough information to try to properly reproduce I think.
One last question:

Did you index the entire hackernews dataset?
Do you observe the bug for any keyword or only specific ones?

In particular, can you test a keyword that happens early in the alphabet, like "ace"?

And thank you so much for your patience and effort!

I did not find a change in schema. This might be a bug in tantivy's sstable based dictionary. I see that you set the raw tokenizer on the text field which I assume can be large.
There might a strange edge case hitting on super large tokens that breaks our sstable dictionary in some way. I have enough information to try to properly reproduce I think.
One last question:

Did you index the entire hackernews dataset?
Yes. I did ingest complete hackernews data.
Do you observe the bug for any keyword or only specific ones?

In particular, can you test a keyword that happens early in the alphabet, like "ace"?
I tested with "ace" and "google", I am getting the same error.
heena@Clickhouse1:~/quickwit-v0.2.1$ export QW_CONFIG=./config/quickwit.yaml heena@Clickhouse1:~/quickwit-v0.2.1$ curl "http://<IP_ADDRESS>:7280/api/v1/hackernews_5/search/stream?query=ace&outputFormat=csv&fastField=id" curl: (18) transfer closed with outstanding read data remaining heena@Clickhouse1:~/quickwit-v0.2.1$ curl "http://<IP_ADDRESS>:7280/api/v1/hackernews_5/search/stream?query=google&outputFormat=csv&fastField=id" curl: (18) transfer closed with outstanding read data remaining heena@Clickhouse1:~/quickwit-v0.2.1$

@fulmicoton , Let me try  if I can share the hackernews data.I am not sure if I can add this much huge file in GitHub,
I downloaded the data from  https://github.com/ClickHouse/ClickHouse/issues/29693#issuecomment-933114791 and converted from native format to JSON using clickhouse and then feed it to Quickwit.
I

Sharing 01G26NHMCV1BAP61AS006H7A75.split would be the most helpful actually.

@HeenaBansal2009 I am able to reproduce ! Thank you. I'll let you know when it is fixed.

@fulmicoton Great ! Thanks , I am waiting to hear back .
Thanks for your patience and support here .

@HeenaBansal2009 In the meanwhile, I was able to identify that the problem is related to the "raw" tokenizer in on the text field.
You probably want the default tokenizer there.
    - name: text
      type: text
      record: position

This will require you to reindex your dataset.

Ok the bug is fully understood. The sstable index was not added to the hotdirectory because we have a filter that filter out chunks larger than 10megabytes to be added.
Because of the raw tokenizer, the split had an unusual number of unique tokens, with address that were unusually large.
The slice containing the stable index was therefore dismissed.

@fulmicoton , Great.  I am keeping eye on the fix .
However , if this situation arises when text has raw tokenenizer, If I change the tokenizer type for text field , do you think this can help with the error  and I will be able to use curl ?
Currently, my task is to measure the performance of Quickwit with large set of data against my proprietary setup?

@HeenaBansal2009
I am not sure I understand.
The bug is... an actual  bug and it will be fixed in Quickwit 0.3 which is meant to be released next week.
We take bugs very seriously :)
Now it triggers only when we have large splits with very large tokens which is most likely not something useful.
For a text with several words in it, that is in average longer than  30 characters, you will always want to use a tokenizer.
The raw tokenizer is more for small field like a user id (c3ke8sa0A), or a severity (INFO, WARNING, etc).
For your performance experiment, do you want us to have a look at the schema and make sure that everything looks fine?

name: payload
type: text
tokenizer: raw
record: position
stored: false

is not good for you. tokenizer: raw means that the entire text is emitted as a single token. That is why the bug triggered.
You want to set tokenizer: default instead.

For instance @HeenaBansal2009
version: 0

index_id:  hackernews_5

doc_mapping:
  store_source: false
  field_mappings:
    - name: id
      type: u64
      fast: true
    - name: type
      type: text
      tokenizer: raw
    - name: by
      type: text
      tokenizer: raw
      record: position
    - name: text
      type: text
      record: position
      tokenizer: default
    - name: url
      type: text
      tokenizer: default
      record: position
    - name: title
      type: text
      tokenizer: default
      record: position

search_settings:
  default_search_fields: [title,by,text]

Oops I replied too fast.
Yes you should probably tokenize your text.
You are correct, with the default tokenizer the IP will have to be searched as a phrase query which is not great.
raw is simply not an option. With raw you will return 0 document in your search.
Ideally we should work on having a tokenizer that suits your use case.
We have a ticket related to that: #1143
In the meanwhile can you give search with position a try?
Also if you care a lot about performance or if you never plan to target a specific field, it might be faster to concatenate by,title, and body and treat them as a single field.

Thanks @fulmicoton  for your inputs. Yes absolutely #1143 will  match our criteria for FT search.
I am looking forward for the fix.

@fulmicoton , I must say Quickwit data ingestion is pretty fast than quickwit. I am impressed . :-)
A quick question:
After indexing json file into quickwit , I see the metrics like 'Indexed 8649189 documents in 9.80min. '
I can see only no. of documents indexed. Is there nay way , I can find how many rows were inserted in to quickwit  , So that I can prepare the performance metrics according to number of rows inserted across speed.

I'm not sure to understand what you mean by "rows" but in Quickwit, you have several data structures where we store information:

we have a doc store which is a row-oriented storage (document id => document stored values)
we have a columnar storage called fastfield where we store the document field values in a continuous manner
we have an inverted index (very simplistic view is term -> term info -> list of doc IDs that contains the term)

As for indexing speed, we have seen an ingestion throughput of 40 MB/s on small servers (4vCPU) with local ssd.

Like for ES , We got the metrics like :
28737558/28737558[1:23:43<00:00/5721.98docs/s] "index_time_in_millis": 28737558,(38m:12s)
Similarly ,  I am looking for QuickWit, Quickwit insert the same data file in almost 1/10 times lesser time.
I see the metrics on console in terms of docs(which can be different as per inmplenetation/partition of data).
I am looking for no. of rows/records inserted.
Like in database : we have no of rows inserted to no of records available in datafile.

Mmm, maybe a command like describe index is what you are looking for? See the docs here: https://quickwit.io/docs/reference/cli#index-describe

In Quickwit jargon, 1 doc = 1 document = 1 record = 1 row.
quickwit index ingest outputs the number of docs ingested for the file.
quickwit index describe outputs the total number of docs in the index.
Does that help?


File Name and Year Search

Need file name search and year search options
from year to to year
assume the file name is associated with a tag or a year tag for each note

Good news, this is already possible. Please see this wiki article and also the docs for Whoosh (the package used for search).


Linux 3.13 9pfs

Upgrade to linux 3.13 kernel, and also add 9PFS support to allow folder sharing with DIOD (github.com/kainz/diod, my port from chaos's DIOD 9PFS server)
Also, enable make_iso.sh to use a custom docker-bin if you want to build a boot2docker using docker HEAD or whatnot.

@kainz I'm going to close this, we've not moved on it, sorry.
If you, or someone else wants to add doid & 9PFS, please make a much more minimal PR :)


Improved convertReflectValueToType

Updated Interrupt tests
Added Interface check to DeleteExpr

Codecov Report

Merging #140 into master will increase coverage by 0.18%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
+ Coverage   72.03%   72.21%   +0.18%
==========================================
  Files          40       40
  Lines        2750     2750
==========================================
+ Hits         1981     1986       +5
+ Misses        624      618       -6
- Partials      145      146       +1

Impacted Files
Coverage Δ

vm/vmExprFunction.go
71.02% <100%> (ø)
:arrow_up:

vm/vmExpr.go
66.79% <100%> (+0.12%)
:arrow_up:

vm/vmConvertToX.go
55.38% <63.63%> (+6.13%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4b5492...b8c7b00. Read the comment docs.

Merge fixed

merged https://github.com/mattn/anko/pull/142

@mattn
Could you please let me know why some of this PR was dropped? Or if there is a general change on how you would like PRs to be done, opened up a issue about it.

Please do not feel wrong. I just want to separate PRs since I prefer that one PR is fix one issue. This is useful to find out what was wrong. I developed anko as I feel as. But now anko seems to be depended on some products. So I decided this. Please don't feel wrong meaning.

No problem, happy to make PRs smaller when possible. Maybe next time it would be possible to share about the change ahead of time to make it easier to adapt?

Everyone is great


Add notify to "Copy zone files"

What is missing?
When ansible change zone file, not restart coredns
Why do we need it?
When ansible change zone file, not restart coredns
Environment

Role version:
0.3.2

Ansible version information:
ansible --version
ansible 2.9.4 config file = None configured module search path = [u'/mnt/data/home/backups-ansible/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /mnt/data/home/backups-ansible/venv-ansible-2.9.4/lib/python2.7/site-packages/ansible executable location = /mnt/data/home/backups-ansible/venv-ansible-2.9.4/bin/ansible python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

Anything else we need to know?:

Why do you want to restart coredns? If the SOA changes, coredns will automatically load the new zone file.

Yes, this seems like it's already correct.


[Kernel] change benchmark script so that result can be directly used; tune moe kernel in A100/H100 with tp=2,4,8

cc @WoosukKwon

Thanks for the refactorings! While you are touching this code, one thing that would be wonderful to do is to keep track of the timings for the best configuration. This could e.g. be done by writing them to a separate file. This would allow you to decide if a new configuration is better than the old one.
Also note that running the script as-is will likely not produce optimal results in some settings, since there is a bunch of parameter pruning going on at the moment (e.g. for the batch size). Sometimes it is important to look at the values found and then expand the search space if it runs into the boundaries :)

@pcmoritz  This manual kernel tuning is kind of temporary. Going forward, we plan to use triton.autotune to automatically tune these configs. So we don't need to invest too much time here.

@pcmoritz This manual kernel tuning is kind of temporary. Going forward, we plan to use triton.autotune to automatically tune these configs. So we don't need to invest too much time here.

In my experience triton.autotune is far too slow to be useful (unless the configs have already been run / are cached) :)

command: python benchmarks/benchmark_throughput.py --model=mistralai/Mixtral-8x7B-Instruct-v0.1 --input-len 1000 --output-len 50 -tp 2
H100 GPU:

benchmark
no config
w/o this PR
w/ PR

tp=2
11.73 requests/s, 12313.00 tokens/s
14.11 requests/s, 14816.20 tokens/s
15.69 requests/s, 16477.71 tokens/s

tp=4
18.07 requests/s, 18974.30 tokens/s
same as no config
22.06 requests/s, 23166.51 tokens/s

tp=8
24.80 requests/s, 26039.89 tokens/s
same as no config
28.17 requests/s, 29577.75 tokens/s

A100 GPU: TODO (don't have 8*A100 GPU at hand now)
@WoosukKwon benchmarking results are quite promising!

In my experience triton.autotune is far too slow to be useful (unless the configs have already been run / are cached) :)

Will definitely try to cache tuned configs!

@youkaichao Awesome! Could you 1) update the PR with the current main and 2) fix the lint error by running ./format.sh? You will have to run pip install -r requirements-dev.txt before it.

@WoosukKwon lint is good now 👌


Added Gefion Gymnasium. by Johan Uhrskov-Bendixsen

Love all of your products!

@johanub Pull request merged. Thank you for your warm words, I really hope that you will enjoy them further. Shall you have any questions regarding JetBrains tools, please email us at<EMAIL_ADDRESS>Have a great year of 2020!


Why is migrations commented out?

The migration task is commented out and not up to date. I currently have:
namespace :deploy do
  task :migrate do
    invoke 'laravel:artisan', 'migrate'
  end
end

should it be updated with the code above?

Same to me. I guess something should be updated in the gems repository. The version 0.0.2 is too old: https://rubygems.org/gems/capistrano-laravel

Can the gem be updated already? The version still being distributed contains the commented out version of this code.

Sorry for the late response. I'll be helping maintain this plugin for the foreseeable future. If any of these issues are still persisting (all of the tickets are quite old), please let me know and I will do what I can to try and help fix things.

@ikari7789 I ran into this issue today. I tried running  invoke 'laravel:artisan', 'migrate' and I got an error that it had already been invoked. Also when I use the build in migration method it says Application in Production -- command cancelled

@mcblum PR #20 should have fixed this. I'll release a minor version update shortly with the fix. Additionally, there is a laravel:migrate_db task which will run migrations.