zero accuracy on mmlu_generative

Hi thanks for providing such wonderful evaluation toolkit.
I was wondering why evaluation on mmlu_generative returns 0 accuracy whenever what models I try (pythia, qwen).
I understand it as a generative version of mmlu, it can be used to evaluate base/instruct model and match the model's output to a formatted target answer ""{{['(A)', '(B)', '(C)', '(D)'][answer]}}""
My command:
python3 -m accelerate.commands.launch --num_processes 8 --main_process_port 12399 lm_eval \
    --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks mmlu_generative \
    --batch_size 32 \
    --log_samples \
    --output_path ./logs/

Results:
hf (pretrained=EleutherAI/pythia-160m,revision=step100000,dtype=float), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 32
|                 Tasks                 |Version|Filter|n-shot|  Metric   |   |Value|   |Stderr|
|---------------------------------------|------:|------|-----:|-----------|---|----:|---|-----:|
|mmlu (generative)                      |      2|none  |      |exact_match|↑  |    0|±  |     0|
|  - formal_logic                       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_european_history       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_us_history             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_world_history          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - international_law                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - jurisprudence                      |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - logical_fallacies                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - moral_disputes                     |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - moral_scenarios                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - philosophy                         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - prehistory                         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_law                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - world_religions                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - business_ethics                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - clinical_knowledge                 |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_medicine                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - global_facts                       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - human_aging                        |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - management                         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - marketing                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - medical_genetics                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - miscellaneous                      |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - nutrition                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_accounting            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_medicine              |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - virology                           |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - econometrics                       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_geography              |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_government_and_politics|      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_macroeconomics         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_microeconomics         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_psychology             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - human_sexuality                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_psychology            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - public_relations                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - security_studies                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - sociology                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - us_foreign_policy                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - abstract_algebra                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - anatomy                            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - astronomy                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_biology                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_chemistry                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_computer_science           |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_mathematics                |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_physics                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - computer_security                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - conceptual_physics                 |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - electrical_engineering             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - elementary_mathematics             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_biology                |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_chemistry              |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_computer_science       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_mathematics            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_physics                |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_statistics             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - machine_learning                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|

|     Groups      |Version|Filter|n-shot|  Metric   |   |Value|   |Stderr|
|-----------------|------:|------|------|-----------|---|----:|---|-----:|
|mmlu (generative)|      2|none  |      |exact_match|↑  |    0|±  |     0|

I would look at the generations in the samples file, and also add some fewshots to the context (say --num_fewshot 5) to prompt the model with the desired format.  Might have a bit more luck but pythia-160m is probably too small to be capable of cohesive generations.

I think it's pretty weird, and it may not related to in-context learning. I also evaluate Qwen/Qwen2-0.5B, it's also 0-acc on mmlu_generative.
And I tested on mmlu_pro which is also a generative task, and it have normal accuracy.
hf (pretrained=Qwen/Qwen2-0.5B-Instruct), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|       Tasks       |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-------------------|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|gpqa_main_zeroshot |      1|none          |     0|acc        |↑  |0.2857|±  |0.0214|
|                   |       |none          |     0|acc_norm   |↑  |0.2857|±  |0.0214|
|mmlu_pro           |      1|custom-extract|      |exact_match|↑  |0.1444|±  |0.0032|
| - biology         |      0|custom-extract|     5|exact_match|↑  |0.2483|±  |0.0161|
| - business        |      0|custom-extract|     5|exact_match|↑  |0.1166|±  |0.0114|
| - chemistry       |      0|custom-extract|     5|exact_match|↑  |0.1025|±  |0.0090|
| - computer_science|      0|custom-extract|     5|exact_match|↑  |0.1195|±  |0.0160|
| - economics       |      0|custom-extract|     5|exact_match|↑  |0.1979|±  |0.0137|
| - engineering     |      0|custom-extract|     5|exact_match|↑  |0.0918|±  |0.0093|
| - health          |      0|custom-extract|     5|exact_match|↑  |0.1467|±  |0.0124|
| - history         |      0|custom-extract|     5|exact_match|↑  |0.1706|±  |0.0193|
| - law             |      0|custom-extract|     5|exact_match|↑  |0.1317|±  |0.0102|
| - math            |      0|custom-extract|     5|exact_match|↑  |0.1288|±  |0.0091|
| - other           |      0|custom-extract|     5|exact_match|↑  |0.1591|±  |0.0120|
| - philosophy      |      0|custom-extract|     5|exact_match|↑  |0.1423|±  |0.0157|
| - physics         |      0|custom-extract|     5|exact_match|↑  |0.1101|±  |0.0087|
| - psychology      |      0|custom-extract|     5|exact_match|↑  |0.2268|±  |0.0148|

| Groups |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|--------|------:|--------------|------|-----------|---|-----:|---|-----:|
|mmlu_pro|      1|custom-extract|      |exact_match|↑  |0.1444|±  |0.0032|

Qwen2-0.5B-Instruct on mmlu_generative.
hf (pretrained=Qwen/Qwen2-0.5B-Instruct), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|                 Tasks                 |Version|Filter|n-shot|  Metric   |   |Value|   |Stderr|
|---------------------------------------|------:|------|-----:|-----------|---|----:|---|-----:|
|mmlu (generative)                      |      2|none  |      |exact_match|↑  |    0|±  |     0|
|  - formal_logic                       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_european_history       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_us_history             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_world_history          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - international_law                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - jurisprudence                      |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - logical_fallacies                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - moral_disputes                     |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - moral_scenarios                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - philosophy                         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - prehistory                         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_law                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - world_religions                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - business_ethics                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - clinical_knowledge                 |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_medicine                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - global_facts                       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - human_aging                        |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - management                         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - marketing                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - medical_genetics                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - miscellaneous                      |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - nutrition                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_accounting            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_medicine              |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - virology                           |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - econometrics                       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_geography              |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_government_and_politics|      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_macroeconomics         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_microeconomics         |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_psychology             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - human_sexuality                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - professional_psychology            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - public_relations                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - security_studies                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - sociology                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - us_foreign_policy                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - abstract_algebra                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - anatomy                            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - astronomy                          |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_biology                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_chemistry                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_computer_science           |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_mathematics                |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - college_physics                    |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - computer_security                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - conceptual_physics                 |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - electrical_engineering             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - elementary_mathematics             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_biology                |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_chemistry              |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_computer_science       |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_mathematics            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_physics                |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - high_school_statistics             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|  - machine_learning                   |      2|none  |     0|exact_match|↑  |    0|±  |     0|

|     Groups      |Version|Filter|n-shot|  Metric   |   |Value|   |Stderr|
|-----------------|------:|------|------|-----------|---|----:|---|-----:|
|mmlu (generative)|      2|none  |      |exact_match|↑  |    0|±  |     0|

I'll take a look! My guess is a bug in the answer extraction

Hello, I also have this error while using the mmlu_generative task to benchmark the llama3 model.
Command:
python3 main.py \
	--model hf \
	--model_args pretrained=model-path\
	--tasks mmlu_humanities_generative \
	--limit 3 \
	--output_path output/ \
	--write_out

Result:
|           Tasks            |Version|Filter|n-shot|  Metric   |   |Value|   |Stderr|
|----------------------------|------:|------|-----:|-----------|---|----:|---|-----:|
|formal_logic                |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|high_school_european_history|      2|none  |     0|exact_match|↑  |    0|±  |     0|
|high_school_us_history      |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|high_school_world_history   |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|international_law           |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|jurisprudence               |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|logical_fallacies           |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|moral_disputes              |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|moral_scenarios             |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|philosophy                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|prehistory                  |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|professional_law            |      2|none  |     0|exact_match|↑  |    0|±  |     0|
|world_religions             |      2|none  |     0|exact_match|↑  |    0|±  |     0|

I also try to dump some intermediate result after add some log info:
a) the prompt input text: add print log for the generate_until API in lm_eval/models/huggingface.py
The following are multiple choice questions (with answers) about world religions.

Which of the following plays the most significant role in forming a child's political views?
A. The geographical area in which the child grows up
B. The child's family
C. The media to which the child is exposed
D. The child's religion
Answer:

b) LLM response from self._model_generate:
The child's religion

It seems the response result looks normal, but the value of exact_match from the final result table is always 0.
Could you plase help to take a look? Thanks

Hello
I have been able to solve this.
I had only change the exact match to this:
def exact_match(gold, pred=None):

if pred is None and isinstance(gold, list):
if len(gold) != 2:
raise ValueError("If passing a single list argument, it must contain
exactly two elements.")
gold, pred = gold
gold = str(gold).strip().upper()
pred = str(pred).strip()
if not pred:
print("Warning: pred is empty")
return 0.0
pred_first_char = pred[0].upper()
value = 1.0 if gold == pred_first_char else 0.0
return value

and I used the exact match her
dataset_path: hails/mmlu_no_train # a copy of cais/mmlu with no
auxiliary_train split
test_split: test
fewshot_split: dev
fewshot_config:
sampler: first_n
output_type: generate_until
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC.
{{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_target: "{{['A', 'B', 'C', 'D'][answer]}}"
generation_kwargs:
until:

""
"\n"
metric_list:
metric: !function utils.exact_match
aggregation: mean
higher_is_better: true
metadata:
version: 2.0
dataset_kwargs:
trust_remote_code: true

Let me know if you have any other questions.
Best
Aisha.
On Thu, 31 Oct 2024 at 11:24, yshi @.***> wrote:

Hello, I also have this error while using the mmlu_generative task to
benchmark the llama3 model.
Command:
python3 main.py
--model hf
--model_args pretrained=model-path
--tasks mmlu_humanities_generative
--limit 3
--output_path output/
--write_out
Result:

Tasks
Version
Filter
n-shot
Metric

Value

Stderr

formal_logic
2
none
0
exact_match
↑
0
±
0

high_school_european_history
2
none
0
exact_match
↑
0
±
0

high_school_us_history
2
none
0
exact_match
↑
0
±
0

high_school_world_history
2
none
0
exact_match
↑
0
±
0

international_law
2
none
0
exact_match
↑
0
±
0

jurisprudence
2
none
0
exact_match
↑
0
±
0

logical_fallacies
2
none
0
exact_match
↑
0
±
0

moral_disputes
2
none
0
exact_match
↑
0
±
0

moral_scenarios
2
none
0
exact_match
↑
0
±
0

philosophy
2
none
0
exact_match
↑
0
±
0

prehistory
2
none
0
exact_match
↑
0
±
0

professional_law
2
none
0
exact_match
↑
0
±
0

world_religions
2
none
0
exact_match
↑
0
±
0

I also try to dump some intermediate result after add some log info:
a) the prompt input text: add print log for the generate_until API in
lm_eval/models/huggingface.py
The following are multiple choice questions (with answers) about world religions.
Which of the following plays the most significant role in forming a child's political views?
A. The geographical area in which the child grows up
B. The child's family
C. The media to which the child is exposed
D. The child's religion
Answer:
b) LLM response from self._model_generate:
The child's religion
It seems the response result looks normal, but the value of exact_match
from the final result table is always 0.
Could you plase help to take a look? Thanks
—
Reply to this email directly, view it on GitHub
https://github.com/EleutherAI/lm-evaluation-harness/issues/2279#issuecomment-2449299632,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AK3TI7DCFNY5IFSLKV56N7TZ6HSM7AVCNFSM6AAAAABNV5XDY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZGI4TSNRTGI
.
You are receiving this because you commented.Message ID:
@.***>

--
DISCLAIMER: The contents of this email and any attachments are
confidential. They are intended for the named recipient(s) only. If you
have received this email by mistake, please notify the sender immediately
and you are herewith notified that the contents are legally privileged and
that you do not have permission to disclose the contents to anyone, make
copies thereof, retain or distribute or act upon it by any means,
electronically, digitally or in print. The views expressed in this
communication may be of a personal nature and not be representative of
AIMS-NEI and/or any of its Centres or Initiatives.

It is a bug in the extraction filtering. Take a look at the this log:
{"doc_id": 9, "doc": {"question": "According to Kant, morality requires us to:", "subject": "philosophy", "choices": ["perform the action that leads to the greatest total happiness.", "act only on maxims that we can will to become universal laws.", "behave only in such a way as a perfectly virtuous person would behave.", "place the interests of others above the interests of ourselves."], "answer": 1}, "target": "B", "arguments": {"gen_args_0": {"arg_0": "The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAccording to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:\nA. pleasure.\nB. happiness.\nC. good.\nD. virtue.\nAnswer: C\n\nAccording to d'Holbach, people always act according to _____.\nA. free choices\nB. dictates of the soul\nC. necessary natural laws\nD. undetermined will\nAnswer: C\n\nAccording to Kant, morality requires us to:\nA. perform the action that leads to the greatest total happiness.\nB. act only on maxims that we can will to become universal laws.\nC. behave only in such a way as a perfectly virtuous person would behave.\nD. place the interests of others above the interests of ourselves.\nAnswer:", "arg_1": {"until": ["</s>", "\n"]}}}, "resps": [[" B"]], "filtered_resps": [" B"], "doc_hash": "c5177394044574b9c8f03867fc2e5db56e8e8904af717f33f6701af2f62c4b17", "prompt_hash": "18cd89493222e9a9fe80fd0b2beaf39dffc9abe61ff3abeb1ad50d9d33ac731c", "target_hash": "df7e70e5021544f4834bbee64a9e3789febc4be81470df629cad6ddb03320a5c", "exact_match": 0.0}

it returns "exact_match": 0.0 because "filtered_resps": [" B"], is not equal to "target": "B",, note the initial space in the filtered answer, this is a normal issue, and I also observed it in BBH.
If we modify the task and templates like this:

files and changes
- `_mmlu.yaml`
group: mmlu_generative
group_alias: mmlu (generative)
task:
  - group: stem
    task:
      - mmlu_stem_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: True
        filter_list: get_response
  - group: other
    task:
      - mmlu_other_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: True
        filter_list: get_response
  - group: social sciences
    task:
      - mmlu_social_sciences_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: True
        filter_list: get_response
  - group: humanities
    task:
      - mmlu_humanities_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: True
        filter_list: get_response
aggregate_metric_list:
  - aggregation: mean
    metric: exact_match
    weight_by_size: True
    filter_list: get_response
metadata:
  version: 2

_default_template_yaml

dataset_path: hails/mmlu_no_train # a copy of `cais/mmlu` with no auxiliary_train split
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: generate_until
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_target: "{{['A', 'B', 'C', 'D'][answer]}}"
generation_kwargs:
  until:
    - "</s>"
    - "\n"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
filter_list:
  - name: get_response
    filter:
      # Filter everything after the first break line
      - function: "regex"
        regex_pattern: "^(.*?)(?=\\n|$)"
      # Remove leading white spaces
      - function: remove_whitespace
      # function to ignore right white spaces or line breaks
      - function: "regex"
        regex_pattern: "^(.*?)\\s*$"
metadata:
  version: 2.0
dataset_kwargs:
  trust_remote_code: true

We will get the expected result:
{"doc_id": 9, "doc": {"question": "According to Kant, morality requires us to:", "subject": "philosophy", "choices": ["perform the action that leads to the greatest total happiness.", "act only on maxims that we can will to become universal laws.", "behave only in such a way as a perfectly virtuous person would behave.", "place the interests of others above the interests of ourselves."], "answer": 1}, "target": "B", "arguments": {"gen_args_0": {"arg_0": "The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAccording to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:\nA. pleasure.\nB. happiness.\nC. good.\nD. virtue.\nAnswer: C\n\nAccording to d'Holbach, people always act according to _____.\nA. free choices\nB. dictates of the soul\nC. necessary natural laws\nD. undetermined will\nAnswer: C\n\nAccording to Kant, morality requires us to:\nA. perform the action that leads to the greatest total happiness.\nB. act only on maxims that we can will to become universal laws.\nC. behave only in such a way as a perfectly virtuous person would behave.\nD. place the interests of others above the interests of ourselves.\nAnswer:", "arg_1": {"until": ["</s>", "\n"]}}}, "resps": [[" B"]], "filtered_resps": [["B"]], "doc_hash": "c5177394044574b9c8f03867fc2e5db56e8e8904af717f33f6701af2f62c4b17", "prompt_hash": "18cd89493222e9a9fe80fd0b2beaf39dffc9abe61ff3abeb1ad50d9d33ac731c", "target_hash": "df7e70e5021544f4834bbee64a9e3789febc4be81470df629cad6ddb03320a5c", "exact_match": 1.0}

see "exact_match": 1.0 at the end of the line.
I tested this on Qwen2.5-32B-Instruct-AWQ (only 50 samples)
The accuracy changed from all zeros to:
|      Groups      |Version|   Filter   |n-shot|  Metric   |   |Value |   |Stderr|
|------------------|-------|------------|------|-----------|---|-----:|---|-----:|
|mmlu (generative) |      2|get_response|      |exact_match|↑  |0.8351|±  |0.0067|
| - humanities     |    N/A|get_response|      |exact_match|↑  |0.8523|±  |0.0136|
| - other          |    N/A|get_response|      |exact_match|↑  |0.8231|±  |0.0144|
| - social sciences|    N/A|get_response|      |exact_match|↑  |0.8700|±  |0.0132|
| - stem           |    N/A|get_response|      |exact_match|↑  |0.8095|±  |0.0122|

This is the same problem I observed in BBH, I'm planning on creaiting a PR later

Hi RawthiL
Thanks for pointing out the missing config for the YAML file. But there are some differences in the output sequence of our model, and here is "State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed" after applying the above patch for the filter config.
we can see the output matches the context of D, but the exact_match is equal to 0 since the response after the filter is not equal to "D". Do you have any experience with this special response for the filter?
Thanks
{"doc_id": 0, "doc": {"question": "Which of the following best describes the balance the Supreme Court has struck between the establishment clause and the free-exercise clause?", "subject": "high_school_government_and_politics", "choices": ["Freedom of speech is protected except in certain situations, such as yelling \"fire\" in a crowded theater.", "Once a church has been recognized by the federal government, its tax-exempt status can never be revoked.", "Once Congress has created an administrative agency, that agency can be dissolved only by a constitutional amendment.", "State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."], "answer": 3}, "target": "D", "arguments": {"gen_args_0": {"arg_0": "The following are multiple choice questions (with answers) about human aging.\n\nWhich of the following best describes the balance the Supreme Court has struck between the establishment clause and the free-exercise clause?\nA. Freedom of speech is protected except in certain situations, such as yelling \"fire\" in a crowded theater.\nB. Once a church has been recognized by the federal government, its tax-exempt status can never be revoked.\nC. Once Congress has created an administrative agency, that agency can be dissolved only by a constitutional amendment.\nD. State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed.\nAnswer:", "arg_1": {"until": ["</s>", "\n"]}}}, "resps": [[" State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."]], "filtered_resps": ["State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."], "doc_hash": "8f63cebd5269df80a7f6386afb6ea7266a908ffe6b72f431cf962d8dc3948358", "prompt_hash": "f63bb19b3a6c11a40c8939643328509dfd97d1b172f25a68894559a9689ba51d", "target_hash": "3f39d5c348e5b79d06e842c114e6cc571583bbf44e4b0ebfda1a01ec05745d43", "exact_match": 0.0}
```

{"doc_id": 0, "doc": {"question": "Which of the following best describes the balance the Supreme Court has struck between the establishment clause and the free-exercise clause?", "subject": "high_school_government_and_politics", "choices": ["Freedom of speech is protected except in certain situations, such as yelling "fire" in a crowded theater.", "Once a church has been recognized by the federal government, its tax-exempt status can never be revoked.", "Once Congress has created an administrative agency, that agency can be dissolved only by a constitutional amendment.", "State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."], "answer": 3}, "target": "D", "arguments": {"gen_args_0": {"arg_0": "The following are multiple choice questions (with answers) about human aging.\n\nWhich of the following best describes the balance the Supreme Court has struck between the establishment clause and the free-exercise clause?\nA. Freedom of speech is protected except in certain situations, such as yelling "fire" in a crowded theater.\nB. Once a church has been recognized by the federal government, its tax-exempt status can never be revoked.\nC. Once Congress has created an administrative agency, that agency can be dissolved only by a constitutional amendment.\nD. State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed.\nAnswer:", "arg_1": {"until": ["", "\n"]}}}, "resps": [[" State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."]], "filtered_resps": ["State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."], "doc_hash": "8f63cebd5269df80a7f6386afb6ea7266a908ffe6b72f431cf962d8dc3948358", "prompt_hash": "f63bb19b3a6c11a40c8939643328509dfd97d1b172f25a68894559a9689ba51d", "target_hash": "3f39d5c348e5b79d06e842c114e6cc571583bbf44e4b0ebfda1a01ec05745d43", "exact_match": 0.0}

It looks like you are doing zero-shot (presenting no examples prior asking the question), this results in the model not being conditioned to respond with a letter (instead an explicit response) and hence the exact match fails.
There is no way to solve that with an exact-match, you will need to create a new test definition for zero shot and probable code a different metric (like a quasi-exact-match).
If there is no important reason for you to use zero-shot, I would suggest you to add --num_fewshots 3.

{"doc_id": 0, "doc": {"question": "Which of the following best describes the balance the Supreme Court has struck between the establishment clause and the free-exercise clause?", "subject": "high_school_government_and_politics", "choices": ["Freedom of speech is protected except in certain situations, such as yelling "fire" in a crowded theater.", "Once a church has been recognized by the federal government, its tax-exempt status can never be revoked.", "Once Congress has created an administrative agency, that agency can be dissolved only by a constitutional amendment.", "State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."], "answer": 3}, "target": "D", "arguments": {"gen_args_0": {"arg_0": "The following are multiple choice questions (with answers) about human aging.\n\nWhich of the following best describes the balance the Supreme Court has struck between the establishment clause and the free-exercise clause?\nA. Freedom of speech is protected except in certain situations, such as yelling "fire" in a crowded theater.\nB. Once a church has been recognized by the federal government, its tax-exempt status can never be revoked.\nC. Once Congress has created an administrative agency, that agency can be dissolved only by a constitutional amendment.\nD. State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed.\nAnswer:", "arg_1": {"until": ["", "\n"]}}}, "resps": [[" State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."]], "filtered_resps": ["State-sponsored prayer during school hours is prohibited, but voluntary prayer by student groups before school is allowed."], "doc_hash": "8f63cebd5269df80a7f6386afb6ea7266a908ffe6b72f431cf962d8dc3948358", "prompt_hash": "f63bb19b3a6c11a40c8939643328509dfd97d1b172f25a68894559a9689ba51d", "target_hash": "3f39d5c348e5b79d06e842c114e6cc571583bbf44e4b0ebfda1a01ec05745d43", "exact_match": 0.0}

It looks like you are doing zero-shot (presenting no examples prior asking the question), this results in the model not being conditioned to respond with a letter (instead an explicit response) and hence the exact match fails. There is no way to solve that with an exact-match, you will need to create a new test definition for zero shot and probable code a different metric (like a quasi-exact-match). If there is no important reason for you to use zero-shot, I would suggest you to add --num_fewshots 3.

Got it, many thanks for your help.


Document opApply as an alias to a function template instance

Bolpat reported this on 2024-06-26T16:29:41Z
Transferred from https://issues.dlang.org/show_bug.cgi?id=24633
CC List

Bolpat

Description
When `opApply` is an alias to an explicit function template instance, the explicit instantiation is used for overload resolution against potentially many `opApply` overloads and it is used to infer the types of `foreach` variables, however, the delegate created by the compiler in the `foreach` body lowering is passed to the function template uninstantiated. By this mechanism, when overloading `opApply`s based on the number of intended `foreach` variables, type inference is possible **and** function attributes (e.g. `@safe`) will be inferred.

Example:
```d
struct A
{
    int opApply(scope int delegate(long) body) => body(42);
}
struct B
{
    int opApply(Body)(scope Body body) => body(42);
}
struct C
{
    int opApplyImpl(Body)(scope Body body) => body(42);
    alias opApply = opApplyImpl!(int delegate(long));
}
void main() @nogc nothrow pure @safe
{
    // Error: `@nogc` function cannot call non-@nogc function `A.opApply`
    // Error: `pure` function cannot call impure function `A.opApply`
    // Error: `@safe` function cannot call `@system` function `A.opApply`
    // Error: function `A.opApply` is not `nothrow`
    foreach (x; A()) { }

    // Error: cannot infer type for `foreach` variable `x`
    foreach (x; B()) { }

    // Good:
    foreach (x; C())
    {
        static assert(is(typeof(x) == long));
        assert(x == 42);
    }
}
```

This behavior is great and should be specified as intended.

dlang-bot commented on 2024-06-26T16:49:30Z
@Bolpat created dlang/dlang.org pull request #3859 "Specify `opApply` as an alias to a function template instance" fixing this issue:

- Fix Bugzilla Issues 23666, 17953, 23116, and 24633

https://github.com/dlang/dlang.org/pull/3859


attachMediaStream on ChromeBeta mobile(android) is returning error

When I call:
attachMediaStream(element, '');
or
attachMediaStream(element, null);
on Chrome Beta 49 on Android 5.1.1 I have the error:

deps.js:13307 TypeError: Failed to execute 'createObjectURL' on 'URL': No function was found that matched the signature provided.
at TypeError (native)
at HTMLVideoElement.Object.defineProperty.set (https://<IP_ADDRESS>:3000/js/deps.js:76464:26)
at attachMediaStream (https://<IP_ADDRESS>:3000/js/deps.js:76887:25)

Any thoughts?

You should provide a MediaStream as a 2nd argument.

FYI we plan to remove this method shortly.


Update Set-PnPSite.md

Small typo in code example 5. Should be LogoFilePath instead of SiteLogoPath.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sharepoint/sharepoint-ps/sharepoint-pnp/Set-PnPSite.md
:white_check_mark:Succeeded
View (sharepoint-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sharepoint/sharepoint-ps/sharepoint-pnp/Set-PnPSite.md
:white_check_mark:Succeeded
View (sharepoint-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@expiscornovus - thanks for catch and fix, much appreciated!


Changed a wallet in a multisig - now there are no transactions or btc

Hello!
I'm not sure if Sparrow is simply not connecting as I can't get any Wallet loading happening on the lower screen - but if so, how do I make that happen? I've restarted Sparrow and restarted my computer with no results. My node is up and running fine.
I switched one wallet out for another one, using the same seed phrase and then everything just disappeared. Please help!
Thanks

Sparrow wasn't connecting. It's still not recognizing my new wallet, but everything is present.


assets:precompile not working with Ruby 3

When I run 'rails assets:precompile', I receive the following error:
rails assets:precompile --trace
** Invoke assets:precompile (first_time)
** Invoke assets:environment (first_time)
** Execute assets:environment
** Invoke environment (first_time)
** Execute environment
** Invoke yarn:install (first_time)
** Execute yarn:install
yarn install v1.22.15
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 0.06s.
** Execute assets:precompile
rails aborted!
NoMethodError: undefined method `length' for nil:NilClass
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size/seekable_io_reader.rb:21:in `chunk'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size/chunky_reader.rb:25:in `[]'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size.rb:164:in `block in size_of_jpeg'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size.rb:163:in `loop'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size.rb:163:in `size_of_jpeg'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size.rb:52:in `block in initialize'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size/reader.rb:18:in `block in open'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size/reader.rb:18:in `open'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size/reader.rb:18:in `open'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size/reader.rb:18:in `open'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size.rb:50:in `initialize'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size.rb:35:in `new'
/usr/local/bundle/gems/image_size-3.0.0/lib/image_size.rb:35:in `path'
/usr/local/bundle/gems/image_optim-0.31.1/lib/image_optim/image_meta.rb:9:in `format_for_path'
/usr/local/bundle/gems/image_optim-0.31.1/lib/image_optim/path.rb:68:in `image_format'
/usr/local/bundle/gems/image_optim-0.31.1/lib/image_optim.rb:109:in `workers_for_image'
/usr/local/bundle/gems/image_optim-0.31.1/lib/image_optim.rb:116:in `optimize_image'
/usr/local/bundle/gems/image_optim-0.31.1/lib/image_optim.rb:159:in `block in optimize_image_data'
/usr/local/bundle/gems/fspath-3.1.2/lib/fspath.rb:34:in `open'
/usr/local/bundle/gems/fspath-3.1.2/lib/fspath.rb:64:in `temp_file'
/usr/local/bundle/gems/image_optim-0.31.1/lib/image_optim.rb:154:in `optimize_image_data'
/usr/local/bundle/gems/image_optim_rails-0.4.3/lib/image_optim/railtie.rb:66:in `optimize_image_data'
/usr/local/bundle/gems/image_optim_rails-0.4.3/lib/image_optim/railtie.rb:38:in `block (2 levels) in <class:Railtie>'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:84:in `call_processor'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:66:in `block in call_processors'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:65:in `reverse_each'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:65:in `call_processors'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:182:in `load_from_unloaded'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:59:in `block in load'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:337:in `fetch_asset_from_dependency_cache'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:43:in `load'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/cached_environment.rb:44:in `load'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/bundle.rb:32:in `block in call'
/usr/local/lib/ruby/3.0.0/set.rb:344:in `each_key'
/usr/local/lib/ruby/3.0.0/set.rb:344:in `each'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/bundle.rb:31:in `call'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:84:in `call_processor'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:66:in `block in call_processors'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:65:in `reverse_each'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/processor_utils.rb:65:in `call_processors'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:182:in `load_from_unloaded'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:59:in `block in load'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:337:in `fetch_asset_from_dependency_cache'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/loader.rb:43:in `load'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/cached_environment.rb:44:in `load'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/base.rb:81:in `find_asset'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/base.rb:88:in `find_all_linked_assets'
/usr/local/bundle/gems/sprockets-4.0.2/lib/sprockets/manifest.rb:125:in `block (2 levels) in find'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19:in `execute'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/promise.rb:564:in `block in realize'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:363:in `run_task'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `block (3 levels) in create_worker'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:335:in `loop'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:335:in `block (2 levels) in create_worker'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `catch'
/usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block in create_worker'

When I tested precompile without this gem, there was no problem
[Environment Details]
docker image: ruby:3.0.2-buster
Rails <IP_ADDRESS>
sprockets 4

Please update image_size (to 3.0.1), that should fix the problem. And please reopen the issue if not.


winston.handleExceptions submits 'false' to papertrail.

Hi,
I'm trying to use winston-papertrail to log nodejs/express unhandled exceptions but they show up as "false" in papertrail.
Code snippet:
var winston = require('winston');
winston.handleExceptions(
  new winston.transports.Papertrail({
        host: 'xxx',
        port: xxx,
        hostname: 'xxx',
        inlineMeta: true
    })
);

Then I throw an exception without catching it, like this:
throw Error('boo');

After spending some time looking into it I could find out that it looks like "level" is being passed as undefined into:
Papertrail.prototype.sendMessage = function (hostname, program, level, message) {

I have this issue too, did you find how to fix it?

I have the same issue.

I have the same issue. Is there a workaround?

There is a way to work around this, here is my winston-papertrail handleExceptions code block.
var paperTrailLogger = new winston.transports.Papertrail(papertrailConfig);
paperTrailLogger.exceptionsLevel = 'error';
winston.handleExceptions(paperTrailLogger)

The Papertrail constructor function doesn't set exceptionsLevel so the default transport exception handling function ends up sending undefined down to Papertrail.log function.

would also be solved by #34

+1 though, @brocksamson solves my problem without having to fork the repo:)
var log       = new winston.Logger()
  , transport = new transportClass( transportConfig.config )

transport.exceptionsLevel = 'error'
log.handleExceptions( transport )

// undefined -> no config needed, true -> is already initialized
log.add( transport, undefined, true )

Note that a complete exception handling solution also requires the code changes from #44.  The code above will handle uncaught exceptions because the base transport class from winston creates the serialized representation of the object.  This will not work however:
try{
    throw new Exception("Guns are for sissies.");
}
catch(ex){
    log.error(ex);
}

See #42 for an explanation of why this occurs.

@brocksamson pull request #44 is pending for that issue

@brocksamson - ugh, you're right. I'm actually not even getting the "false" messages now. Hopefully this will be fixed soon..........

Also seeing this, any chance of updating this transport?

@jareware (and anyone else who wants this): check out and subscribe to #34

Good to know, thanks!

If you're reading this message, we'd love if you can pull the current master of winston-papertrail and run it. We'll wait at least 2 weeks to see whether any issues come up from wider use of this and one other change. If you encounter problems with master which don't occur in the current release, please open a new GitHub Issue with the transcript and reference #34.

This is live on NPM in version 1.0.3.


Unexpected token: name (onClickOutside) [./~/react-onclickoutside/index.js:92,0]

I ran into a problem.
I'm using react-create-app and during build it causes error:
SyntaxError : Unexpected token: name (onClickOutside) [./~/react-onclickoutside/index.js:92,0]
Have you ever had similar issues or know how to handle it?

This HOC is a pure ES6 implementation as of v6.x - if you need ES5 code you can either use 5.x or below, or set up your build system so that it does ES5 conversion for you, at whatever is the most logical point in your build chain.
Use
yarn add<EMAIL_ADDRESS>

v6.1.0 has been published and should address your problem. Please let me know if you're still running into this after udpating.

@Pomax Thanks, it has fixed the issue for me!


:state="false" on BFormCheckbox

Describe the bug
Shouldn't the first checkbox on https://bootstrap-vue-next.github.io/bootstrap-vue-next/docs/components/form-checkbox.html#contextual-state-and-validation-example be red since the state prop is set to false?
I have a required checkbox that I want to highlight in red under certain circumstances.
This code didn't produce a checkbox at all on StackBlitz for some reason:
<BFormCheckbox :state="false">Should be red</BFormCheckbox>

Reproduction
N/A
Used Package Manager
npm
Validations

[X] Have tested with the latest version. This is still alpha version and sometime things change rapidly.
[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

@jonkri - Thanks, the individual checkbox === false behavior is a bug. I have #2019 up to address that. I'm not reproing the no checkbox issue. Can you add a link to your stackblitz?

0.23.3

@dwgray: Thanks! False alarm on the stackblitz problem... I didn't realize I had to import the components in that context. 😅


Normalize assert calls in tests

Some unusual calls were forgotten in the preview PR

Q
A

Type
improvement

BC Break
no

Fixed issues
#2441

Summary
Normalizes assert*() calls in tests

Thanks @hunomina!


Sonarqube fixes2

Merging in various fixes suggested by SonarQube

Codecov Report

Merging #427 into master will increase coverage by 0.73%.
The diff coverage is 88.46%.

@@             Coverage Diff              @@
##             master     #427      +/-   ##
============================================
+ Coverage     84.71%   85.45%   +0.73%
+ Complexity     2283     2279       -4
============================================
  Files           159      159
  Lines          6858     6834      -24
  Branches       1022     1014       -8
============================================
+ Hits           5810     5840      +30
+ Misses          726      672      -54
  Partials        322      322

Impacted Files
Coverage Δ
Complexity Δ

...cecenter/xenon/adaptors/webdav/WebdavLocation.java
71.42% <ø> (ø)
4 <0> (ø)
:arrow_down:

...tors/gridengine/GridEngineSchedulerConnection.java
82.53% <0%> (ø)
56 <0> (ø)
:arrow_down:

...er/xenon/adaptors/generic/DirectoryStreamBase.java
94.73% <100%> (ø)
11 <4> (ø)
:arrow_down:

...ciencecenter/xenon/adaptors/ssh/OpenSSHConfig.java
91.12% <100%> (ø)
21 <1> (ø)
:arrow_down:

.../esciencecenter/xenon/engine/util/JobExecutor.java
76.47% <100%> (-2.59%)
37 <5> (-5)

...nl/esciencecenter/xenon/engine/util/JobQueues.java
94.63% <100%> (+1.46%)
63 <1> (ø)
:arrow_down:

.../main/java/nl/esciencecenter/xenon/util/Utils.java
67.47% <100%> (+3.34%)
71 <4> (+6)
:arrow_up:

...non/adaptors/torque/TorqueSchedulerConnection.java
74.52% <100%> (ø)
51 <0> (ø)
:arrow_down:

.../xenon/adaptors/scripting/SchedulerConnection.java
83.87% <100%> (-0.69%)
44 <6> (-5)

...nter/xenon/adaptors/ssh/SshMultiplexedSession.java
74.3% <100%> (-1.39%)
33 <0> (-1)

... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8564334...df08dbd. Read the comment docs.


Requesting for downloading the app "designing holograms" on HoloLens 2

Describe the problem
Hi. I want to download the app "designing holograms" on HoloLens 2. Because I can't open the Microsoft store due to the Web Proxy, so I download the app from microsoft website, but it tolds me that "It looks like you don’t have any applicable device(s) linked to your Microsoft account. To install, sign in on a device with your Microsoft account." However, I always sign in on my HoloLens 2 with my account.

Describe the solution you'd like
Use the app "designing holograms" on HoloLens 2 or my computer.
Describe alternatives you've considered
No.

Hey @962557346, this is outside the scope of the MRTK team.  I recommend filing this bug on Feedback Hub.
You can find Feedback bug by selecting the Start  button, and then typing or selecting Feedback Hub.
Full instructions here: https://support.microsoft.com/en-us/windows/send-feedback-to-microsoft-with-the-feedback-hub-app-f59187f8-8739-22d6-ba93-f66612949332


Animation Capture tracks don't work anymore

Godot version
4.2 dev6
System information
Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1060 (NVIDIA; <IP_ADDRESS>3) - Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (8 Threads)
Issue description
Capture tracks no longer blend the properties. The final value is reached immediately.
https://github.com/godotengine/godot/assets/2223172/cf401330-7f51-492d-8b9b-37c0465ff531
Steps to reproduce

Add AnimationPlayer
Add Animation1
Add Animation2 with capture track
Play Animation1 then Animation2

Minimal reproduction project
bgu.tscn.txt
works correctly in dev5, but not 6

This issue is just come to the surface of a latent bug same as https://github.com/godotengine/godot/issues/82949.
https://github.com/godotengine/godot/issues/48432
https://github.com/godotengine/godot/issues/55110

I can confirm this on 4.2.beta da0b1eb12 (Linux). This can be reproduced in https://github.com/godotengine/tps-demo by aiming with the right-mouse button (the camera smoothing transition is broken and snaps instantly to the final position).
This broke between 4.2.dev5 and 4.2.dev6. I bisected the regression to 1b95827d3ef244de322b0c16deb49fefe48ed1a1.

So from what I can see, https://github.com/godotengine/godot/pull/80813 removed every bit of implementation of the capture update mode. For the most part it used to work like continuous mode, and this is still true because many checks in code are NOT UPDATE_DISCRETE. However, AnimationPlayer used to have a dedicated branch for the capture mode in _animation_process_animation. AnimationTree had no special code.
Now there is no special code in the AnimationMixer either, so the mode basically doesn't exist.

As was reported a long time ago in https://github.com/godotengine/godot/issues/55110, it would be quite difficult to support it correctly at this stage, since AnimationTree cannot recognize that playback has resumed in the middle of a playback. Also I don't have any idea yet if it is possible to include such information about the start of playback in the https://github.com/godotengine/godot-proposals/issues/8083 refactor.

I sent a proposal https://github.com/godotengine/godot-proposals/issues/8513.

I just want to say: Ever since I learned about Capture Mode in the AnimationPlayer, years ago, I use it a lot.
I planned on using 4.2 for a new project and not having Capture Mode, or a dynamic animation mode for the AnimationPlayer would probably make me reconsider.

Please don't bump issues without contributing significant new information. Use the :+1: reaction button on the first post instead.

@AThousandShips
"I need this feature which has been removed for my upcoming project" I cannot express with a 👍 reaction button. A thumbs-up is just a general approval and can mean all kings of things.
You know I have been with this community for years, testing and reporting issues regularly. So "Please don't bump issues with useless reminders."

You needing this does not constitute important information for the bug, it does not affect solving it or furthering the process
Please, since you have been part of the community for years you should understand this

I've unhidden your original comment to stop this pointless debate, but I would say that @AThousandShips is correct here. We've already identified the issue, we've established its importance and the animation team is looking for a solution. Stating that you personally need this resolved doesn't actually help in any way at this point.
Upvoting, however, does. As it's a quick and easy way for us to notice what affects most users rather than specific users.
In any case, I don't think there is more to be said about this, so let's leave the comments to the discussion about possible solutions.

Glad to know the issue is tracked. In the meantime if anyone is trying to use this system to interpolate extremely simple "animations" with very few values, like a moving or rotating platform: look into tweens. Uglier, but you should get the job done with them.

Given that this has been broken for several months at this point, and has probably caused many cumulative hours of frustration with Godot, it would be a good idea to:

Document the current breakage so future users don't also have to wind up here after time wasted debugging, then googling. Perhaps something like https://github.com/godotengine/godot-docs/pull/8764
Remove "capture" from the drop-down of animation mode options


Move user's configuration to a separate Makefile

In order to avoid Git conflicts between existing complicated targets of current Makefile and the custom user's code while Template upgrade.
Plus, change the documentation to teach users not to edit the "our" Makefile but only "theirs".
--
"their" Makefile: `Makefile`
--
include neuro.mk

DATA_DIR?=data
CONFIG_DIR?=config
CODE_DIR?={{cookiecutter.code_directory}}
NOTEBOOKS_DIR?=notebooks
RESULTS_DIR?=results
...

--
"our" Makefile: `neuro.mk`
--

##### COMMANDS #####

APT?=apt-get -qq
PIP?=pip install --progress-bar=off -U --no-cache-dir
NEURO?=neuro

ifeq (${TRAIN_STREAM_LOGS}, yes)
	TRAIN_WAIT_START_OPTION=--wait-start --detach
else
	TRAIN_WAIT_START_OPTION=
endif
...

not urgent for Jan2020 iteration.

outdated


assert-twice-that.eo:49-52: Make this object decorates...

The puzzle 167-8884c4e0 from #167 has to be resolved:
https://github.com/objectionary/eo-hamcrest/blob/97e3ced2bb37763d5ca0712b5273bc9ade13a3fc/src/main/eo/org/eolang/hamcrest/assert-twice-that.eo#L49-L52
The puzzle was created by @Graur on 24-Nov-22.
Estimate: 45 minutes,  role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

@graur please pay attention to this new issue.


Swift4 support?

Would welcome a Swift4 compatible version. Any chance of this on the horizon?

FYI: For others looking to run under Swift 4.0
__
I went through the SwiftForm Xcode9 migration process from Swift 3.1 to Swift 4.0. This occurs when you launch the project, and Xcode9 detects your legendary code.
Many "fix-it's" appeared, and I clicked them all - many effectively prepends all #selector referenced functions with an @objc
For example, in FormSegmentedControl.swift the function:
internal func valueChanged(_ sender: UISegmentedControl) {
}
becomes:
@objc internal func valueChanged(_ sender: UISegmentedControl) {
}
Some hard numeric references were also required to change from
    titleLabel.setContentCompressionResistancePriority(500, for: .horizontal)

to:
titleLabel.setContentCompressionResistancePriority(UILayoutPriority(rawValue: 500), for: .horizontal)

Once I achieved a clean compile I then ran and received the error:
Terminating app due to uncaught exception 'NSUnknownKeyException', reason: '[<SwiftForms.FormSegmentedControlCell 0x105822400> valueForUndefinedKey:]: this class is not key value coding-compliant for the key segmentedControl.'
So, either the blind changes were problematic, or something fundamental changed preventing SwiftForms from functioning in Swift 4.
The WWDC What's New in Foundation video references changes to Key/Value observing.
https://developer.apple.com/videos/play/wwdc2017/212/
Uncertain if this is part of the issue. Xcode9 now allows one to not compile under Swift by clicking the SwiftForms Pod target and selecting Build with Swift 3.2.

Xcode9 now allows you to selectively decide the compiler to use to compile each project target. In this case, I selected the Pods project in Xcode, then clicked on the SwiftForms target, then under Build Settings set "Swift Language Version" to Swift 3.2
I was able to compile my project and no longer received the key/value error described above. So this appears to be a near-term workaround, though a true Swift4 compatible version would be welcomed.
Thanks again for this nice library.

A new Swift4 compatible branch has been uploaded. Hope it will be helpful. :)

Thank you very much.
On Mon, Jun 26, 2017 at 3:18 AM, Miguel Ángel Ortuño <
<EMAIL_ADDRESS>wrote:

A new Swift4 compatible branch has been uploaded. Hope it will be helpful.
:)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ortuman/SwiftForms/issues/158#issuecomment-311020064,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIFFGfvMoExJXP-sa6OSRChPm6HfUj9Nks5sH4VdgaJpZM4Nzihm
.

For others which may want this branch, use the following line in your Podfile
pod 'SwiftForms', :git => 'https://github.com/ortuman/SwiftForms.git', :branch => 'Swift4'

downloaded the beta 2 iOS release. Using the Swift4 branch, everything compiles clean.
Though, when I display my form I now receive the following message, again related to the key value expection error. This was gone for a bit, though back with a vengeance. Any suggestions appreciated.
2017-07-04 14:42:01.302930-0700 [717:453953] [UIFocus] Focus system disabled
2017-07-04 14:42:01.314447-0700 [717:453953] *** Terminating app due to uncaught exception 'NSUnknownKeyException', reason: '[<SwiftForms.FormSegmentedControlCell 0x1030d4200> valueForUndefinedKey:]: this class is not key value coding-compliant for the key segmentedControl.'


Support breaking change detection for modular API & models layer

EngSys and Automation Categorization

[ ] Parity with existing tooling in Track2
[x] New functionality

api layer and rest layer are removed. modular layer is simply copied from top layer, so no need to support sub layers for now.


(Spring Security) Input sanitation and validation

Inputs need to be validated (are they in the format and range we expect?) and sanitized (are we sure they do not contain some code that will be executed somewhere else?).
date and version strings are validated by a strict regex, and that validation will stop any code from getting through. By "id"s are not validated or cleaned. And maybe not subset jsons before posting/putting either.

Input validation we will define while developing, I'll express my vision in separate issues,
The main idea is that

some subset values are not allowed, and
some combination of values in different fields are not allowed, as well as
some actions with some values are not allowed based on its previous state or state of other subset fields.
in addition, some values will be influenced by changes done in other fields.


Issue when use with react-native-camera.

React-native-camera android scan QR is too slow, so i use this library to scan.
But i have a problem:

a required meta-data tag in.....

Any helps? I really love your lib.

What does this even mean? Could you provide some more info? :)

@ducpt2 how to fix? what's two lib?


Document book format and structure

Once the API is kind of stable we should document it :D

And we should remove this folder https://github.com/pythonecuador/lira/tree/master/design


[FLINK-26546] Extract Observer Interface

This PR is mean to extract the Observer interface to simplify the observe logic. The main logic is aligned to the Reconciler interface

cc @morhidi @gyfora @wangyang0918

Thanks @Aitozi, looks good at a first glance, I will review this in detail tomorrow morning :)

Just a personal taste, compared to inheritance, I prefer composition. The problem of inheritance is that we do not know which methods in BaseReconciler and BaseObserver will be overridden by a subclass. Currently, it is not a major problem since we have very few methods in the interface.
Note: I do not mean we have to change in this PR.

@wangyang0918 Thanks for sharing your insights. IMO, one of the benefits of inherited BaseXX class is that it can provide the template ability and leave it implemented by the subclass. The composition seems can not achieve the same effect.
For example, in the BaseReconciler#cleanup provide the common template part, and the both subclass only have to handle how to shutdown the corresponding component.

I also like the BaseXX approach, I think now we have a nice clean separation of observers, and reconcilers :)


Check for libEGL.dll

This is effectively all we need to support Angle.

@tomaka I strongly suggest to use logging. There is nearly not enough variety of error messages, and even if you extend that - would be great to have some extra information on what glutin is doing. This PR's logic is a good example of why logging would be helpful.
Anyhow, I just removed the info! usage.

Usually I think that using log reveals that the design of the library is wrong.
But glutin and other similar libraries try to "just work" by being a big magic instead of having a very precise behavior and exposing every single possible situation, so I guess it makes sense to use logging for them.

Looks like winit broke even though it passed CI recently.


Add jmh test suite to track performance improvements/regressions

At the moment, we (Netflix) are doing performance evaluation of Hystrix in an indirect way - running our production systems with different versions of Hystrix and noting any deltas.
This is a coarse approach that over-emphasizes some paths through Hystrix and under-emphasizes others.  Using jmh would be a more stable (and public) way to track performance over time.

Closing this now.  Will track new performance work in a separate ticket


Add CI

Add issue templates
Add CI pipeline
Add dependabot
Set additional clippy lints

Hey @liamwh, excuse the ridiculous delay. I'm back.
I presume this is still mergable?
Thanks

Indeed. There's more we could do here but this is already a great starting point. Welcome back!

Swell. Thanks for your contributions!


Dataset split issue

In your description, "For data division, 3701 (index < 18505), 200 (18505 <= index < 19505) and 200 (index >= 19505) identities are utilized for training, validation and testing, respectively (Marked by item 'split' in the json file)."
BUT I found in the json file, the train, test, val split are not the same as you said.
Following are what I got:

subset
ids
images
captions

train
3901
19505
39010

test
100
500
1000

val
100
500
1000

So which spilt is correct?

Thanks for your interest in our work. The split with 3701 (index < 18505), 200 (18505 <= index < 19505) and 200 (index >= 19505) identities for training, validation and test sets, respectively, is the correct one. Please try to re-download the newest version of our RSTPReid dataset, which we have checked again.

Thank you for your quick reply. I re-download the dataset and the issue is solved.


Load existing media element

Adds the ability to load an existing HTML5  or  element with loadExistingMediaElement method in wavesurfer.js. Also adds a loadElt method to wavesurfer's mediaelement.js file.
Usage:
init wavesurfer with backend: 'MediaElement'.
 wavesurfer.util.ajax({
    responseType: 'json',
    url: waveformPath // path to .json file with precomputed waveform data
  }).
  on('success', function (data) {
      wavesurfer.loadExistingMediaElement( htmlElement, data);
  });

What is the best way to document this? Should I add a line to  docs/methods.html in the gh-pages branch and an example?
If anyone has suggestions let me know and I'll happily update the PR.
( re https://github.com/katspaugh/wavesurfer.js/issues/698 )

Coverage decreased (-0.2%) to 4.587% when pulling 2805b724c40048d4a4f7a67427829117381ed84f on therewasaguy:load-existing-media-elt into 26657a8d0f0df56d6fd46ea472313c4eb5036e4c on katspaugh:master.

What is the best way to document this? Should I add a line to docs/methods.html in the gh-pages branch and an example?

Yeah, sure. An example with an existing video element would be nice, since it's asked about a lot.
Thank you very much!

Thanks for your quick feedback, @katspaugh! All good points, and I'm sorry I should have updated some of these before sending the PR.
I will update this all soon, and I'll add a demo with video.

Coverage decreased (-0.08%) to 4.667% when pulling 48ffdde1b743ffeaa69cfabd1f9c68a7e6435da1 on therewasaguy:load-existing-media-elt into 622322d35d9be2a3f1d4bd44e7ac91788cacae6e on katspaugh:master.

Updated and added an example using a <video>.
While consolidating the code, I was thinking... why not could just use load for existing media elements, rather than create a separate method? If the first argument is a media element, load handles it correctly. Does that sound like a good approach?

Thanks!

Does that sound like a good approach?

Yep, sounds like a good idea!

Coverage decreased (-0.08%) to 4.667% when pulling b3e811280903cb9a43eda0cfdf2a413186f2ac31 on therewasaguy:load-existing-media-elt into 622322d35d9be2a3f1d4bd44e7ac91788cacae6e on katspaugh:master.

Coverage decreased (-0.08%) to 4.667% when pulling b202eb1313d3c47f62f54cd5e76e0d508a252ee8 on therewasaguy:load-existing-media-elt into 622322d35d9be2a3f1d4bd44e7ac91788cacae6e on katspaugh:master.

Thanks a lot, @therewasaguy!


Variables are not being displayed

I'm using clangd and for both C/C++ the variables are not being displayed in the symbols window.
Everything LSP related works correctly (diagnostics errors and warnings).
.
I cannot unfold the main symbol with l. The keybindings are the default, as shown in README.
Is this something I misconfigured, or is it language server dependent, i.e. some ls support variable symbols while others don't?

This looks like all the information that Clangd provides. I think the result will be similar in VSCode with the Microsoft C/C++ plugin. Anyway, this is sufficient for navigation.


Android Nearby in Brazil is incosistent

Hi @scottjenson. We have 200 beacons deployed in a Brazilian Proximity Marketing Project and nearby is incosistent. Sometimes works normally and sometimes not. Should it silently notifies when a supposed smartphone turn on (local+bluetooth) gets on a perimeter bluetooth? some smartphones properly notifies and others only minutes later...

Hello @lxborges Scott Jenson is not longer involved in the Physical Web. See this comment https://github.com/google/physical-web/issues/931#issuecomment-340843273
Also there are many similar posts to yours here on github and on the Google Group https://groups.google.com/forum/m/#!forum/physical-web-discuss
Please have a look.

Hi @ferencbrachmann . Tks for the info. I have read the blog and noticed that is normal to notify from nearby 5-8 mins. To receive fast, the user should turn off the screen and turn on again or going to nearby app directly. Also, vague phrases or generic in the title (notification message) are not good sometimes. Am I correct?

@lxborges Yes, I've detailed this in the blogpost.

Yes. I have read it. Useful blog and article. Thank you!


Removal of snarky js test from mina

This is one of the step required to fully split snarkyjs and mina repos to preserve healthy dependency relations for both. Test is already implemented in snarkyjs repo where we execute e2e test against real network per push/PR basis, it takes ~10-15 minutes
CI job:  https://github.com/o1-labs/o1js/blob/main/.github/workflows/live-tests.yml
Uses shared actions:  https://github.com/o1-labs/o1js/blob/main/.github/actions/live-tests-shared/action.yml
Sets TEST_TYPE: 'Live integration tests' and USE_CUSTOM_LOCAL_NETWORK: 'true'
And triggers sh run-ci-tests.sh
That in turn executes this test https://github.com/o1-labs/o1js/blob/main/src/examples/zkapps/hello_world/run_live.ts against https://hub.docker.com/r/o1labs/mina-local-network
Quting Serhii, an expert for o1js test architecture:

Having it and more checks after submodule revert I think it is safe to remove snarkyjs test from test_executive since it anyway does similar (or even the same) thing just from mina repo

https://github.com/MinaProtocol/mina/blob/2d427357911c9896cfe6a2e3ce0b5de827bf6363/src/app/test_executive/snarkyjs.ml#L59
https://github.com/o1-labs/o1js/blob/main/tests/integration/simple-zkapp.js

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

Another problem that I have with this PR is that it's only to develop. But rampup, snarkyjs-main and berkeley will need the same changes so starting it here means you can't just merge those upwards

But rampup, snarkyjs-main and berkeley will need the same changes

I don't think we should do it for berkeley or rampup. Those branches are after code freeze and such change can disrupt further RC

In my mind, removing o1js and its test would have been the last task of the submodule refactor

Is i understand from Serhii, this test duplicates o1js tests anyhow so we can get rid of it event now

I don't think we should do it for berkeley or rampup. Those branches are after code freeze and such change can disrupt further RC

Moving o1js out of the Mina repo was in large part motivated by making it easier for other teams, in particular protocol, to land changes. So the change definitely has to go into rampup and berkeley.

In my mind, removing o1js and its test would have been the last task of the submodule refactor

Is i understand from Serhii, this test duplicates o1js tests anyhow so we can get rid of it event now

No, there's no exact equivalent of this test elsewhere currently. o1js tests against a Mina node compiled from either main, berkeley or develop branches. This tests against a Mina node compiled directly from the same branch. So this test here is able to test a change that affects both the Mina node and o1js at the same time, and this aspect is not replicated in any other test.
There are cases where this behaviour is helpful. But they are rare to be honest. So I'm OK with removing the o1js integration test, as long as we don't remove the o1js unit tests yet.

!ci-build-me

!ci-build-me

!ci-build-me


update readme with details about firebase setup

Hi Randy, I've added some notes to the readme about firebase setup. You weren't sure the steps were necessary, so not sure you'll actually want to commit this. I'm mainly doing this as a way to practice collaborating on git.

Thanks Hailey!