provider-aws-s3 bucket examples incorrect

What happened?

Bucket examples produce the following error message:
The Bucket "example-${Rand.RFC1123subdomain}" is invalid: metadata.name: Invalid value: "example-${Rand.RFC1123Subdomain}": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(.a-z0-9?)*')
How can we reproduce it?

Apply one of the examples from here and receive the error.
What environment did it happen in?

Crossplane Version:
Provider Version:
Kubernetes Version:
Kubernetes Distribution:

Those are uptest markers (see https://github.com/upbound/uptest#injecting-dynamic-values-and-datasource).
I agree it should be better documented.


stern with json-pp-logs

Note: this issue was imported from https://github.com/wercker/stern/issues/143, but it was originally created by tansaku...
Has anyone ever managed to get stern working with json-pp-logs? (from https://github.com/kballenegger/rack-json-logs)
I find that it works with kubectl logs, but not with stern and I'm not sure why.  This works fine in terms of reformatting:
kubectl logs -f --tail 100 <pod-name>  | json-log-pp -o -e -v

but not
stern <pod-name>  | json-log-pp -o -e -v

I guess because stern is pre-prending the pod name ... is there a way to have stern avoid printing the pod name prefix?  I can't immediately see how from stern --help
Many thanks in advance

In this case, you can use the --output raw option or the --template flag.


Ecto.Multi.insert_all/5 doesn't autogenerate UUID

Environment

Elixir version (elixir -v): 1.10.3
Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): PostgreSQL 12
Ecto version (mix deps): 3.1
Database adapter and version (mix deps): 0.15.7
Operating system: Windows 10

Current behavior
This will work:
Repo.insert!(
  %Vote{
    user_id: user_id,
    input_id: hd(input_ids)
  },
  prefix: Triplex.to_prefix(tenant)
)

Query generated:

INSERT INTO "localhost"."votes" ("input_id","user_id","id","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5)

This won't:
new_votes =
 [
   %{
     input_id: "2c92c315-b81f-413e-8014-b7a0ab2038a6",
     user_id: "2dd65dcd-9741-4075-943c-c372c8a2ae97"
   }
 ]

Ecto.Multi.new()
|> Ecto.Multi.delete_all(:delete_votes, delete_votes_query, prefix: Triplex.to_prefix(tenant))
|> Ecto.Multi.insert_all(:insert_all, Vote, new_votes, prefix: Triplex.to_prefix(tenant))
|> Repo.transaction()

Error:

** (Postgrex.Error) ERROR 23502 (not_null_violation) null value in column "id" violates not-null constraint
table: votes
column: id

Query generated:

INSERT INTO "localhost"."votes" ("input_id","user_id") VALUES ($1,$2)

I do not understand why Ecto.Multi.insert_all doesn't generate id column like Repo.insert! does.
In the Vote schema we have:
@primary_key {:id, Ecto.UUID, autogenerate: true}
Expected behavior
Expecting the ID to generate.

This is by design. insert_all won't generate ID or timestamps, as it is meant to be low-level. The docs are clearer about this regarding timestamps but not UUID, I will make it clearer. You can call Ecto.UUID.generate or similar. Thanks!

Should I generate IDs and timestamps and use insert_all, or rather iterate over the "votes" and insert them one by one with Repo.insert?
Both have drawbacks, not sure which is best. Which way would you recommend?

But jose see in Repo.insert_all docs:

When a schema module is given, the entries given will be properly dumped before being sent to the database. If the schema contains an autogenerated ID field, it will be handled either at the adapter or the storage layer. However any other autogenerated value, like timestamps, won't be autogenerated when using insert_all/3.

autogenerate is set to true. Doesn't that contradict what you said for the ID?

Please see updated docs. It only applies to the ones generated by the database/adapter and Ecto.UUID isn’t.


allow running SpellCheck manually

Coverage: 91.411%. Remained the same when pulling 5dcb38ffc142bff6681dca0d732d0a2f5e78726b on ranocha-patch-1 into 4ddff297f37371943a4b2939ec031577fa633a48 on main.


Exhibition View: Remove "Meet the Artists"

Remove "Meet the Artists"
Leave only "Discover the Artwork" and "Explore the Tours"

This issue can be closed as it is implemented in the current beta build.


Various improvements to no_std support

Proposed Changes
A collection of general improvements to our no_std support code:

Use ockam_core::compat::mutex instead of cortex_m::interrupt
Make credentials optional -- (this change is specifically to support smaller image sizes for no_std targets)
Various improvements to ockam_executor
Support no_std+alloc builds for ockam_transport_core
Only use cortex_m_semihosting on arm platforms
New feature no_main to control ockam_node_attribute behavior on baremetal platforms

merged https://github.com/ockam-network/ockam/pull/2023 !


fix: Improve chunked upload reliability

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.05%) to 91.94%

Files with Coverage Reduction
New Missed Lines
%

box_sdk_gen/managers/chunked_uploads.py
5
86.82%

Totals

Change from base Build<PHONE_NUMBER>:
-0.05%

Covered Lines:
9970

Relevant Lines:
10844

💛  - Coveralls


Note rendered outside document's bounds

I thought I'd post this seeing the recent round of annotation fixes. This is a regression from PR #4318, at least partial, because prior to the change, the note's color was wrong (yellow).

Added an "InteractiveAnnotation" class to homogenize the annotations' structure (highlighting) and user interactions (for now, used for text and link annotations).

Text annotations:
The appearance (AP) has priority over the icon (Name).
The popup extends horizontally (up to a limit) as well as vertically.
Reduced the title's font size.
The annotation's color (C) is used to color the popup's background.
On top of the mouseover show/hide behavior, a click on the icon will lock the annotation open (for mobile purposes). It can be closed with another click on either the icon or the popup.

An annotation printing is conditioned by its "print" bit
Unsupported annotations are not displayed at all.

@timvandermeij

On a second thought, users can click the annotation, pinning it in the position, and use the horizontal slider to reveal the hidden part of the annotation. Adobe Acrobat Reader moves the annotation to the left when there's not enough space to render the annotation, though.

So to be sure I understand the problem correctly, the only issue here is that the pop-up extends horizontally, causing it to be partially out of view? (The color and position of the icon indeed are the same as in Adobe Acrobat for me.)

Exactly. Actually, the horizontal extension is correct (as opposed to the vertical before PR #4318) -- it's just that it shouldn't extend beyond the edge of the screen, instead the position of the annotation should be adjusted to the left, such that the right edge of the annotation aligns with the edge of the screen.

I noticed that annotation pop-ups can be rendered outside of the pdf document.

Yes, and this issue is already exactly about that.
In the default viewer this should only be a minor inconvenience, since as mentioned in https://github.com/mozilla/pdf.js/issues/6230#issuecomment-122438789 the user should still be able to access such PopupAnnotations just fine in practice.
The reason that this issue hasn't been fixed yet, despite its age, is likely because it's a somewhat minor issue that's probably not trivial to address without a bunch of complexity (since it's easy to imagine various edge-cases).


Fix LiveQuery SSL

This allows LiveQueryServer to work over Parse Sever with SSL

This is the fix described in #271

Just a few style concerns: we're trying to use let/const on all new code.

'var' has updated with 'let' and 'const'

Awesome. I'll see if I can cut a 1.8.5 branch to include this.


On send GA

When is the on send feature expected to GA?

@kloe-cole Thanks for your question. The on-send feature is now available in requirement set 1.8.
Thanks.

@ElizabethSamuel-MSFT  is this feature available and will it also allow the add-in to be published in the marketplace?

@kloe-cole While the feature is now available for use in production add-ins, it is not allowed in AppSource.
Thanks.

@ElizabethSamuel-MSFT is there a timeline of when apps using this feature will be allowed in Appsource?


Flapp Bird Added #22 Resolved

PR Description 📜
Added The Flappy Bird Game
Fixes #22

Mark the task you have completed ✅

[x] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[x] I have added screenshot for website preview in assets/images
[x] I have added entries for my game in main README.md
[x] I have added README.md in my folder
[x] I have added working video of the game in README.md (optional)
[x] I have specified the respective issue number for which I have requested the new game.

Add your screenshots(Optional) 📸

Thank you soo much for contributing to our repository 💗

I Have Increased the play Area

Do You need me to increase it further

Thank you  @AbhishekPSingh07 , for your valuable time and contribution in our GameZone 💗.
It’s our GameZone, so Let’s build this GameZone altogether !!🤝
Hoping to see you soon with another PR again 😇
Wishing you all the best for your journey into Open Source🚀


Modal flickering

Hi, when I open the modal the transition between .modal <-> .is-active leads to flicker effect.
I have to set also the enter class to avoid this issue. Do you have an idea why this is necessary? I try it like in the example. In vue-admin you create the modal in a different way.
:enter-class="enterClass"

According to the changes https://github.com/vuejs/vue/releases
In Vuejs 2.1.7 the transition component was changed.

When fixing #4510 (ensure leave transitions and enter transitions triggered by the same state change are started in the same frame), we had to move the application of the v-enter-active and v-leave-active classes to one frame after the application of v-enter and v-leave.

Example

Thank you. Try vue 2.1.8.


[Bug] When evaluation,  {prediction} in origin_prompt is not replaced with model's response?

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] The bug has not been fixed in the latest version.

Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'GPU 0,1': 'NVIDIA A800-SXM4-80GB',
 'MMEngine': '0.10.4',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.105',
 'OpenCV': '4.10.0',
 'PyTorch': '2.3.0+cu121',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.3.6 (Git Hash '
                              '86e6af5974177e513fd3fee58425e1063e7f1361)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.1\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 8.9.2\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.1, '
                              'CUDNN_VERSION=8.9.2, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DUSE_FBGEMM '
                              '-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-unused-function -Wno-unused-result '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=pedantic '
                              '-Wno-error=old-style-cast -Wno-missing-braces '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, '
                              'USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, '
                              'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, '
                              'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0]',
 'TorchVision': '0.18.1',
 'numpy_random_seed':<PHONE_NUMBER>,
 'opencompass': '0.2.5+',
 'sys.platform': 'linux'}

Reproduces the problem - code/configuration sample
with read_base():
    from ..configs.datasets.subjective.multiround.mtbench101_judge import subjective_datasets

api_meta_template = dict(
    round=[
        dict(role='SYSTEM', api_role='SYSTEM'),
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ]
)
qwen_meta_template = dict(
    round=[
        dict(role="HUMAN", begin='<|im_start|>user\n', end='<|im_end|>\n'),
        dict(role="BOT", begin="<|im_start|>assistant\n", end='<|im_end|>\n', generate=True),
    ],
    eos_token_id=151645,
)
GPU_NUMS = 2

v1_6_path = "xxx"
models = [
    dict(
        abbr='ja_v1_6',
        type=VLLM,
        path=v1_6_path,
        model_kwargs=dict(tensor_parallel_size=GPU_NUMS),
        meta_template=qwen_meta_template,
        max_seq_len=4096,
        max_out_len=4096,
        batch_size=4,
        stop_words=['<|im_end|>'],
        run_cfg=dict(num_gpus=GPU_NUMS, num_procs=1),
    )
]
datasets = [x for x in subjective_datasets if x['abbr'] in ['mtbench101_test', 'mtbench101_ja_test']]

judge_models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4-1106-preview',
    key='sk-xxx',
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=8,
    temperature=0.8,
)]

eval = dict(
    partitioner=dict(type=SubjectiveSizePartitioner, max_task_size=100000, mode='singlescore', models=models, judge_models=judge_models),
    runner=dict(type=LocalRunner, max_num_workers=32, task=dict(type=SubjectiveEvalTask)),
)

summarizer = dict(type=MTBench101Summarizer, judge_type='single')

work_dir = 'outputs/mtbench101/'

Reproduces the problem - command or script
python run.py xxx/eval_subjective_mtbench101.py --debug

Reproduces the problem - error message
这是Evaluation后result目录下的json文件，注意看：
有2个case，第2个case的prediction说The assistant's response is marked as \"{prediction}\" which indicates that there is no actual response provided in the dialogue to evaluate，意思是说origin_prompt中HUMAN说的话里，那个{prediction}并没有被填充（用模型的回答），而奇怪的是，第1个case的prediction是在正常打分，打了最高分10分。
我的问题有2个：

2个case的origin_prompt里，HUMAN说的话里都有1个{prediction}，此处并没有填充为模型的真实回答，这是正常的么？
第1个case打分正常，第2个case说prediction没被填充所以无法打分，为何会有这种差异？

{
    "2": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge ...... Note that only the response of the 'assistant' in the LAST ROUND of conversations is the output of the large language model (the AI assistant) that needs to be evaluated!! You must provide your explanation. After providing your explanation, please show the score by strictly following this format: 'Rating: [[score]]', for example 'Rating: [[6]]'. The DIALGUE need to be judged is in this format: \n *** \n DIALGUE \n ***\n"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: 現在、A、B、C の 3 人がいます。現在、A は B より背が高く、B は C より背が高いことがわかっています。現在、最も背が高いのは誰でしょうか。\n\nAssistant: 与えられた情報に基づくと、A は 3 人の中で最も背が高いです。\n\n Human: 今はあと 2 人、D と E がいます。D は B より背が高く、E は D より背が高いです。現時点で一番背が高いのは誰ですか?\n\nAssistant: 追加情報に基づいて、現時点で最も背の高い人物が誰であるかを判断することはできません。提供された情報には、個人間の相対的な身長が記載されているだけで、A、C、D、E 間の具体的な比較は示されていません。\n\n Human: さて、D が A より高いことはわかっています。現在、最も高いのは誰ですか?\n\nAssistant:  {prediction} \n ***\n\n                    The reference solution is: \n ### xxx（省略） ###\n\n"
            }
        ],
        "prediction": "The assistant's final response is correct. The assistant provided a clear logic for determining who is the tallest among the individuals in question. The response builds on the updated information that D is taller than A, and since A was initially the tallest among A, B, and C, and E is taller than D, E is concluded to be the tallest.\n\nThe reasoning is sound and follows the principles of logic. The assistant correctly integrated the historical dialogue information provided by the user, which is essential for determining the correct answer.\n\nThe assistant's communication is understandable and instructive. It uses the information provided in the conversation effectively to reach the correct conclusion. It also clearly explains the reasoning process, which aids the user's comprehension of how the answer was derived.\n\nRating: [[10]]",
        "gold": {xxx }
    },
    "3": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Judge Prompt，同上"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: ビタミンを摂取すると健康になりますか？\n\nAssistant:  {prediction} \n ***"
            }
        ],
        "prediction": "The assistant's response is marked as \"{prediction}\" which indicates that there is no actual response provided in the dialogue to evaluate. Without a specific response from the assistant, it is impossible to determine whether the assistant would have appropriately used a rhetorical question to clarify the human's query about vitamins and health, or whether it would have provided a detailed and accurate answer. Since there is no actual response from the AI assistant to assess, it cannot be scored based on the given criteria.\n\nRating: [[1]]",
        "gold": {xxx}
    }
}

Other information
No response

感谢您对MT-Bench-101的关注！
针对你的两个问题：

2个case的origin_prompt里，HUMAN说的话里都有1个{prediction}，此处并没有填充为模型的真实回答，这是正常的么？

这不正常，所有的答案都应该被填充。
我这里拉取最新的OpenCompass代码测试，并没有发现异常。 我的测试方式为将models和judge_models都设为最简单的GPT-4-API，然后从原始数据中随便找两条快速测试，运行 python run.py configs/eval_subjective_mtbench101.py --debug
models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=1,
    temperature=0.8,
)]
infer = dict(
    partitioner=dict(type=SizePartitioner, max_task_size=10000),
    runner=dict(
        type=LocalRunner,
        partition='llm_dev2',
        quotatype='auto',
        max_num_workers=32,
        task=dict(type=OpenICLInferTask),
    ),
)
judge_models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=8,
    temperature=0.8,
)]

结果为：
{
    "0": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant:  Given that A is taller than B and B is taller than C, we can infer the following order of height:\n\n- A is taller than B.\n- B is taller than C.\n\nFrom this information, it is clear that A is the tallest among the three. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response is evaluated according to the given criteria:\n\n1. **Accuracy of Answer**: The assistant correctly identifies A as the tallest based on the provided information, which matches the reference solution. \n\n2. **Completeness and Clarity of Reasoning**: The assistant provides a step-by-step explanation, clearly stating that A is taller than B and B is taller than C, leading to the conclusion that A is the tallest. \n\n3. **Integration of Historical Dialogue**: The assistant effectively uses the historical dialogue to derive the correct answer.\n\n4. **Communication and Instruction**: The assistant communicates the solution in a clear and instructive manner, making it easy for the human to understand and follow the logic.\n\nOverall, the assistant's response is accurate, logically structured, and well-communicated.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "1"
        }
    },
    "1": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant:  Given the updated information:\n- A is taller than B.\n- B is taller than C.\n- D is taller than B.\n- E is taller than D.\n\nTo determine the tallest among A, B, C, D, and E, we need to compare A and E, since they are the tallest in their respective sequences.\n\nHowever, we don't have any direct information comparing A and E. Based on the given information alone, we cannot definitively say who is the tallest between A and E.\n\nIf no additional information is provided, we can only conclude that either A or E is the tallest. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response correctly identifies that there is insufficient information to determine definitively who the tallest person is between A and E. It provides a logical and clear explanation for this conclusion by comparing A and E and recognizing that the relative heights between the sequences cannot be determined with the given information.\n\n### Evaluation Criteria:\n\n1. **Accuracy of the Answer:** The assistant's answer aligns with the reference solution in accurately stating that it is impossible to determine the tallest person definitively.\n2. **Reasoning Process:** The assistant provides a sound reasoning process by comparing the heights of individuals within both sequences and acknowledging the lack of direct information between A and E.\n3. **Integration of Historical Dialogue:** The assistant effectively integrates the historical dialogue and considers all the previous information provided.\n4. **Communication:** The explanation is clear, logical, and understandable, which aids in the human’s comprehension of the situation.\n\nGiven the assistant's thoroughness in addressing the problem and the clarity of its reasoning, it meets the criteria very well.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "2"
        }
    },
    "2": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant: Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.\n\n Human: Now, I know that D is higher than A. Who is the highest now?\n\nAssistant:  Given the new piece of information that D is taller than A, and previously knowing that E is taller than D, E is the tallest among all the individuals (A, B, C, D, and E). \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}, {'role': 'user', 'content': 'Now, I know that D is higher than A. Who is the highest now?'}, {'role': 'assistant', 'content': 'Based on the updated information, if D is higher than A and A is already known to be taller than B and B is taller than C, E is higher than D. then E would be the tallest person at the moment.'}] \n ###\n\n"
            }
        ],
        "prediction": "The AI assistant's response accurately identifies that E is the tallest person based on the newly provided information that D is taller than A, and E is taller than D. This matches the reference solution.\n\n1. **Accuracy**: The response correctly concludes that E is the tallest. This is consistent with the reference solution.\n2. **Reasoning Process**: The AI assistant provides a clear and logical reasoning process:\n   - It restates the new information that D is taller than A.\n   - It uses the previous information that E is taller than D to conclude that E is the tallest.\n3. **Integration of Historical Dialogue**: The AI assistant integrates the historical information correctly, considering all given height comparisons.\n4. **Communication**: The explanation is clear and understandable, effectively communicating the solution.\n\nOverall, the assistant's response meets all the evaluation criteria effectively.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "3"
        }
    },

Human结果填充正常，评测prediction结果正常。

第1个case打分正常，第2个case说prediction没被填充所以无法打分，为何会有这种差异？

第1个case，第2个case中，HUMAN说的话里都有1个{prediction}未被填充，两个case都不正常，只不过第1个稍微说了几轮才填充失败，第2个第一轮就填充失败，导致GPT4拒绝回答行为略有差异。
但这个问题不是主要问题，你应该优先解决第一个问题，即为什么不会填充。

Conclusion

由于我这里拉取最新的代码测试无异常，你再检查下是否有使用疏忽？以及你的模型回答是日语，感觉有点奇怪，是否引入了特殊字符导致填充失败？

感谢您对MT-Bench-101的关注！
针对你的两个问题：

2个case的origin_prompt里，HUMAN说的话里都有1个{prediction}，此处并没有填充为模型的真实回答，这是正常的么？

这不正常，所有的答案都应该被填充。 我这里拉取最新的OpenCompass代码测试，并没有发现异常。 我的测试方式为将models和judge_models都设为最简单的GPT-4-API，然后从原始数据中随便找两条快速测试，运行 python run.py configs/eval_subjective_mtbench101.py --debug
models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=1,
    temperature=0.8,
)]
infer = dict(
    partitioner=dict(type=SizePartitioner, max_task_size=10000),
    runner=dict(
        type=LocalRunner,
        partition='llm_dev2',
        quotatype='auto',
        max_num_workers=32,
        task=dict(type=OpenICLInferTask),
    ),
)
judge_models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=8,
    temperature=0.8,
)]

结果为：
{
    "0": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant:  Given that A is taller than B and B is taller than C, we can infer the following order of height:\n\n- A is taller than B.\n- B is taller than C.\n\nFrom this information, it is clear that A is the tallest among the three. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response is evaluated according to the given criteria:\n\n1. **Accuracy of Answer**: The assistant correctly identifies A as the tallest based on the provided information, which matches the reference solution. \n\n2. **Completeness and Clarity of Reasoning**: The assistant provides a step-by-step explanation, clearly stating that A is taller than B and B is taller than C, leading to the conclusion that A is the tallest. \n\n3. **Integration of Historical Dialogue**: The assistant effectively uses the historical dialogue to derive the correct answer.\n\n4. **Communication and Instruction**: The assistant communicates the solution in a clear and instructive manner, making it easy for the human to understand and follow the logic.\n\nOverall, the assistant's response is accurate, logically structured, and well-communicated.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "1"
        }
    },
    "1": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant:  Given the updated information:\n- A is taller than B.\n- B is taller than C.\n- D is taller than B.\n- E is taller than D.\n\nTo determine the tallest among A, B, C, D, and E, we need to compare A and E, since they are the tallest in their respective sequences.\n\nHowever, we don't have any direct information comparing A and E. Based on the given information alone, we cannot definitively say who is the tallest between A and E.\n\nIf no additional information is provided, we can only conclude that either A or E is the tallest. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response correctly identifies that there is insufficient information to determine definitively who the tallest person is between A and E. It provides a logical and clear explanation for this conclusion by comparing A and E and recognizing that the relative heights between the sequences cannot be determined with the given information.\n\n### Evaluation Criteria:\n\n1. **Accuracy of the Answer:** The assistant's answer aligns with the reference solution in accurately stating that it is impossible to determine the tallest person definitively.\n2. **Reasoning Process:** The assistant provides a sound reasoning process by comparing the heights of individuals within both sequences and acknowledging the lack of direct information between A and E.\n3. **Integration of Historical Dialogue:** The assistant effectively integrates the historical dialogue and considers all the previous information provided.\n4. **Communication:** The explanation is clear, logical, and understandable, which aids in the human’s comprehension of the situation.\n\nGiven the assistant's thoroughness in addressing the problem and the clarity of its reasoning, it meets the criteria very well.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "2"
        }
    },
    "2": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant: Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.\n\n Human: Now, I know that D is higher than A. Who is the highest now?\n\nAssistant:  Given the new piece of information that D is taller than A, and previously knowing that E is taller than D, E is the tallest among all the individuals (A, B, C, D, and E). \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}, {'role': 'user', 'content': 'Now, I know that D is higher than A. Who is the highest now?'}, {'role': 'assistant', 'content': 'Based on the updated information, if D is higher than A and A is already known to be taller than B and B is taller than C, E is higher than D. then E would be the tallest person at the moment.'}] \n ###\n\n"
            }
        ],
        "prediction": "The AI assistant's response accurately identifies that E is the tallest person based on the newly provided information that D is taller than A, and E is taller than D. This matches the reference solution.\n\n1. **Accuracy**: The response correctly concludes that E is the tallest. This is consistent with the reference solution.\n2. **Reasoning Process**: The AI assistant provides a clear and logical reasoning process:\n   - It restates the new information that D is taller than A.\n   - It uses the previous information that E is taller than D to conclude that E is the tallest.\n3. **Integration of Historical Dialogue**: The AI assistant integrates the historical information correctly, considering all given height comparisons.\n4. **Communication**: The explanation is clear and understandable, effectively communicating the solution.\n\nOverall, the assistant's response meets all the evaluation criteria effectively.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "3"
        }
    },

Human结果填充正常，评测prediction结果正常。

第1个case打分正常，第2个case说prediction没被填充所以无法打分，为何会有这种差异？

第1个case，第2个case中，HUMAN说的话里都有1个{prediction}未被填充，两个case都不正常，只不过第1个稍微说了几轮才填充失败，第2个第一轮就填充失败，导致GPT4拒绝回答行为略有差异。 但这个问题不是主要问题，你应该优先解决第一个问题，即为什么不会填充。

Conclusion

由于我这里拉取最新的代码测试无异常，你再检查下是否有使用疏忽？以及你的模型回答是日语，感觉有点奇怪，是否引入了特殊字符导致填充失败？

感谢您对MT-Bench-101的关注！
针对你的两个问题：

2个case的origin_prompt里，HUMAN说的话里都有1个{prediction}，此处并没有填充为模型的真实回答，这是正常的么？

这不正常，所有的答案都应该被填充。 我这里拉取最新的OpenCompass代码测试，并没有发现异常。 我的测试方式为将models和judge_models都设为最简单的GPT-4-API，然后从原始数据中随便找两条快速测试，运行 python run.py configs/eval_subjective_mtbench101.py --debug
models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=1,
    temperature=0.8,
)]
infer = dict(
    partitioner=dict(type=SizePartitioner, max_task_size=10000),
    runner=dict(
        type=LocalRunner,
        partition='llm_dev2',
        quotatype='auto',
        max_num_workers=32,
        task=dict(type=OpenICLInferTask),
    ),
)
judge_models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=8,
    temperature=0.8,
)]

结果为：
{
    "0": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant:  Given that A is taller than B and B is taller than C, we can infer the following order of height:\n\n- A is taller than B.\n- B is taller than C.\n\nFrom this information, it is clear that A is the tallest among the three. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response is evaluated according to the given criteria:\n\n1. **Accuracy of Answer**: The assistant correctly identifies A as the tallest based on the provided information, which matches the reference solution. \n\n2. **Completeness and Clarity of Reasoning**: The assistant provides a step-by-step explanation, clearly stating that A is taller than B and B is taller than C, leading to the conclusion that A is the tallest. \n\n3. **Integration of Historical Dialogue**: The assistant effectively uses the historical dialogue to derive the correct answer.\n\n4. **Communication and Instruction**: The assistant communicates the solution in a clear and instructive manner, making it easy for the human to understand and follow the logic.\n\nOverall, the assistant's response is accurate, logically structured, and well-communicated.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "1"
        }
    },
    "1": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant:  Given the updated information:\n- A is taller than B.\n- B is taller than C.\n- D is taller than B.\n- E is taller than D.\n\nTo determine the tallest among A, B, C, D, and E, we need to compare A and E, since they are the tallest in their respective sequences.\n\nHowever, we don't have any direct information comparing A and E. Based on the given information alone, we cannot definitively say who is the tallest between A and E.\n\nIf no additional information is provided, we can only conclude that either A or E is the tallest. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response correctly identifies that there is insufficient information to determine definitively who the tallest person is between A and E. It provides a logical and clear explanation for this conclusion by comparing A and E and recognizing that the relative heights between the sequences cannot be determined with the given information.\n\n### Evaluation Criteria:\n\n1. **Accuracy of the Answer:** The assistant's answer aligns with the reference solution in accurately stating that it is impossible to determine the tallest person definitively.\n2. **Reasoning Process:** The assistant provides a sound reasoning process by comparing the heights of individuals within both sequences and acknowledging the lack of direct information between A and E.\n3. **Integration of Historical Dialogue:** The assistant effectively integrates the historical dialogue and considers all the previous information provided.\n4. **Communication:** The explanation is clear, logical, and understandable, which aids in the human’s comprehension of the situation.\n\nGiven the assistant's thoroughness in addressing the problem and the clarity of its reasoning, it meets the criteria very well.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "2"
        }
    },
    "2": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant: Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.\n\n Human: Now, I know that D is higher than A. Who is the highest now?\n\nAssistant:  Given the new piece of information that D is taller than A, and previously knowing that E is taller than D, E is the tallest among all the individuals (A, B, C, D, and E). \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}, {'role': 'user', 'content': 'Now, I know that D is higher than A. Who is the highest now?'}, {'role': 'assistant', 'content': 'Based on the updated information, if D is higher than A and A is already known to be taller than B and B is taller than C, E is higher than D. then E would be the tallest person at the moment.'}] \n ###\n\n"
            }
        ],
        "prediction": "The AI assistant's response accurately identifies that E is the tallest person based on the newly provided information that D is taller than A, and E is taller than D. This matches the reference solution.\n\n1. **Accuracy**: The response correctly concludes that E is the tallest. This is consistent with the reference solution.\n2. **Reasoning Process**: The AI assistant provides a clear and logical reasoning process:\n   - It restates the new information that D is taller than A.\n   - It uses the previous information that E is taller than D to conclude that E is the tallest.\n3. **Integration of Historical Dialogue**: The AI assistant integrates the historical information correctly, considering all given height comparisons.\n4. **Communication**: The explanation is clear and understandable, effectively communicating the solution.\n\nOverall, the assistant's response meets all the evaluation criteria effectively.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "3"
        }
    },

Human结果填充正常，评测prediction结果正常。

第1个case打分正常，第2个case说prediction没被填充所以无法打分，为何会有这种差异？

第1个case，第2个case中，HUMAN说的话里都有1个{prediction}未被填充，两个case都不正常，只不过第1个稍微说了几轮才填充失败，第2个第一轮就填充失败，导致GPT4拒绝回答行为略有差异。 但这个问题不是主要问题，你应该优先解决第一个问题，即为什么不会填充。

Conclusion

由于我这里拉取最新的代码测试无异常，你再检查下是否有使用疏忽？以及你的模型回答是日语，感觉有点奇怪，是否引入了特殊字符导致填充失败？

感谢您对MT-Bench-101的关注！
针对你的两个问题：

2个case的origin_prompt里，HUMAN说的话里都有1个{prediction}，此处并没有填充为模型的真实回答，这是正常的么？

这不正常，所有的答案都应该被填充。 我这里拉取最新的OpenCompass代码测试，并没有发现异常。 我的测试方式为将models和judge_models都设为最简单的GPT-4-API，然后从原始数据中随便找两条快速测试，运行 python run.py configs/eval_subjective_mtbench101.py --debug
models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=1,
    temperature=0.8,
)]
infer = dict(
    partitioner=dict(type=SizePartitioner, max_task_size=10000),
    runner=dict(
        type=LocalRunner,
        partition='llm_dev2',
        quotatype='auto',
        max_num_workers=32,
        task=dict(type=OpenICLInferTask),
    ),
)
judge_models = [dict(
    abbr='GPT4-Turbo',
    type=OpenAI,
    path='gpt-4o-2024-05-13', # To compare with the official leaderboard, please use gpt-4-1106-preview
    key='',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=16,
    max_out_len=4096,
    max_seq_len=4096,
    batch_size=8,
    temperature=0.8,
)]

结果为：
{
    "0": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant:  Given that A is taller than B and B is taller than C, we can infer the following order of height:\n\n- A is taller than B.\n- B is taller than C.\n\nFrom this information, it is clear that A is the tallest among the three. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response is evaluated according to the given criteria:\n\n1. **Accuracy of Answer**: The assistant correctly identifies A as the tallest based on the provided information, which matches the reference solution. \n\n2. **Completeness and Clarity of Reasoning**: The assistant provides a step-by-step explanation, clearly stating that A is taller than B and B is taller than C, leading to the conclusion that A is the tallest. \n\n3. **Integration of Historical Dialogue**: The assistant effectively uses the historical dialogue to derive the correct answer.\n\n4. **Communication and Instruction**: The assistant communicates the solution in a clear and instructive manner, making it easy for the human to understand and follow the logic.\n\nOverall, the assistant's response is accurate, logically structured, and well-communicated.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "1"
        }
    },
    "1": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant:  Given the updated information:\n- A is taller than B.\n- B is taller than C.\n- D is taller than B.\n- E is taller than D.\n\nTo determine the tallest among A, B, C, D, and E, we need to compare A and E, since they are the tallest in their respective sequences.\n\nHowever, we don't have any direct information comparing A and E. Based on the given information alone, we cannot definitively say who is the tallest between A and E.\n\nIf no additional information is provided, we can only conclude that either A or E is the tallest. \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}] \n ###\n\n"
            }
        ],
        "prediction": "The assistant's response correctly identifies that there is insufficient information to determine definitively who the tallest person is between A and E. It provides a logical and clear explanation for this conclusion by comparing A and E and recognizing that the relative heights between the sequences cannot be determined with the given information.\n\n### Evaluation Criteria:\n\n1. **Accuracy of the Answer:** The assistant's answer aligns with the reference solution in accurately stating that it is impossible to determine the tallest person definitively.\n2. **Reasoning Process:** The assistant provides a sound reasoning process by comparing the heights of individuals within both sequences and acknowledging the lack of direct information between A and E.\n3. **Integration of Historical Dialogue:** The assistant effectively integrates the historical dialogue and considers all the previous information provided.\n4. **Communication:** The explanation is clear, logical, and understandable, which aids in the human’s comprehension of the situation.\n\nGiven the assistant's thoroughness in addressing the problem and the clarity of its reasoning, it meets the criteria very well.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "2"
        }
    },
    "2": {
        "origin_prompt": [
            {
                "role": "SYSTEM",
                "prompt": "Please act as an impartial judge follow this instructions: xxx"
            },
            {
                "role": "HUMAN",
                "prompt": "The dialogue need to be judged is: \n *** \n \n\n Human: Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?\n\nAssistant: Based on the given information, A is the tallest among the three people.\n\n Human: Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?\n\nAssistant: Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.\n\n Human: Now, I know that D is higher than A. Who is the highest now?\n\nAssistant:  Given the new piece of information that D is taller than A, and previously knowing that E is taller than D, E is the tallest among all the individuals (A, B, C, D, and E). \n ***\n\n                    The reference solution is: \n ### \n [{'role': 'user', 'content': 'Now there are three people A, B and C. I currently know that A is taller than B and B is taller than C. Who is the tallest currently?'}, {'role': 'assistant', 'content': 'Based on the given information, A is the tallest among the three people.'}, {'role': 'user', 'content': 'Now there are two more people, D and E. D is higher than B and E is higher than D. Who is the tallest at the moment?'}, {'role': 'assistant', 'content': 'Based on the additional information, it is not possible to determine who the tallest person is at the moment. The given information only states the relative heights between the individuals but does not provide any specific comparison between A, C, D, and E.'}, {'role': 'user', 'content': 'Now, I know that D is higher than A. Who is the highest now?'}, {'role': 'assistant', 'content': 'Based on the updated information, if D is higher than A and A is already known to be taller than B and B is taller than C, E is higher than D. then E would be the tallest person at the moment.'}] \n ###\n\n"
            }
        ],
        "prediction": "The AI assistant's response accurately identifies that E is the tallest person based on the newly provided information that D is taller than A, and E is taller than D. This matches the reference solution.\n\n1. **Accuracy**: The response correctly concludes that E is the tallest. This is consistent with the reference solution.\n2. **Reasoning Process**: The AI assistant provides a clear and logical reasoning process:\n   - It restates the new information that D is taller than A.\n   - It uses the previous information that E is taller than D to conclude that E is the tallest.\n3. **Integration of Historical Dialogue**: The AI assistant integrates the historical information correctly, considering all given height comparisons.\n4. **Communication**: The explanation is clear and understandable, effectively communicating the solution.\n\nOverall, the assistant's response meets all the evaluation criteria effectively.\n\nRating: [[10]]",
        "gold": {
            "multi_id": 1,
            "task": "GR",
            "turn_id": "3"
        }
    },

Human结果填充正常，评测prediction结果正常。

第1个case打分正常，第2个case说prediction没被填充所以无法打分，为何会有这种差异？

第1个case，第2个case中，HUMAN说的话里都有1个{prediction}未被填充，两个case都不正常，只不过第1个稍微说了几轮才填充失败，第2个第一轮就填充失败，导致GPT4拒绝回答行为略有差异。 但这个问题不是主要问题，你应该优先解决第一个问题，即为什么不会填充。

Conclusion

由于我这里拉取最新的代码测试无异常，你再检查下是否有使用疏忽？以及你的模型回答是日语，感觉有点奇怪，是否引入了特殊字符导致填充失败？

拉取最新代码（我的代码其实不旧，是24号上午拉的），其他都不变，包括数据、模型、配置等，然后没问题了。看了下最近2天的commit，感觉跟这个有关：


Rename SliverLogicalParentData.scrollOffset to layoutOffset

This quantity is actually the layoutOffset of the child, not its scroll offset.

@Hixie


vimPlugins.hardhat-nvim: add dependencies

Description of changes
Add plugin dependencies.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Is it possible to add some checks  ? e.g.
    doInstallCheck = true;
    nvimRequireCheck = "hardhat";  # will start nvim with require'hardhat'

are overseer and telescope mandatory dependencies ? I find that a bit strange, if they are optional I would leave them out to minimize potential breakage. Not everyone is using telescope either.

are overseer and telescope mandatory dependencies ? I find that a bit strange, if they are optional I would leave them out to minimize potential breakage. Not everyone is using telescope either.

Actually, only plenary is a real "dependency". Let's keep only this one.


items.sh: return early when no Taps directory

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

Workaround for removing errors output in #14254. Will need API usage for actual completion support, but haven't had chance to look into this.
Haven't tested on HOMEBREW_INSTALL_FROM_API setup, but seemed to work when I changed Taps in following line to invalid directory:
https://github.com/Homebrew/brew/blob/884c4bec14073314ac6e8e9a6a64bc3b26161ab6/Library/Homebrew/items.sh#L21

e.g. changing above to Taps2 and trying without this PR:
❯ brew formulae
find: /opt/homebrew/Library/Taps2: No such file or directory

❯ brew info find: /opt/homebrew/Library/Taps2: No such file or directory
find: /opt/homebrew/Library/Taps2: No such file or directory
brew

with PR:
❯ brew formulae

❯ brew info <TAB>
<NOTHING>

Review period will end on 2022-12-19 at 21:44:38 UTC.

Review period ended.

Thanks again @cho-m!


error log

Hello I have several errors in HA is this normal?
THANKS
2024-04-14 11:54:17.282 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.cupra_request_results (<class 'custom_components.seatconnect.sensor.SeatSensor'>) took 0.481 seconds. Please create a bug report at https://github.com/farfar/homeassistant-seatconnect/issues
2024-04-14 12:06:44.785 WARNING (MainThread) [seatconnect.connection] Could not fetch timers, error: 'NoneType' object has no attribute 'get'
2024-04-14 12:06:44.787 WARNING (MainThread) [seatconnect.connection] Could not fetch position, error: 'NoneType' object has no attribute 'get'
2024-04-14 12:06:44.788 WARNING (MainThread) [seatconnect.connection] Could not fetch charger, error: 'NoneType' object has no attribute 'get'
2024-04-14 12:06:44.790 WARNING (MainThread) [seatconnect.connection] Could not fetch trip statistics, error: 'NoneType' object has no attribute 'get'
2024-04-14 12:06:44.793 WARNING (MainThread) [seatconnect.connection] Could not fetch StoredVehicleDataResponse, error: 'NoneType' object has no attribute 'get'

This code is no longer being maintained by me and requires contributors in order to fix issues.


Add merge queue CI settings

Hey it takes forever to get a bunch of PRs merged. Let's add a queue! 🇬🇧
Additions

Add merge queue CI settings

See https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue#triggering-merge-group-checks-with-github-actions

How do I build docker images for PRs now?


Framework for accelerating compression using QAT

Accelerate compression to provide improved efficiency, scalability and
performance. Intel QuickAssist Technology (Intel QAT) offloading
Compression from CPU which can save cycles, time and space.
As the complexity of applications continues to grow, systems need more
and more computational resources for workloads, including cryptography
and data compression.
The drivers and patches offered here assist application developers to
take advantage of Intel® QuickAssist Integrated Accelerator that is
offered with platforms based on Intel® processors.
To Learn how to use Intel® QuickAssist Technology and run example code,
review our tutorial videos on Intel Developer Zone – Intel® QuickAssist
Technology Technical Getting Started Tutorials -
https://software.intel.com/en-us/networking/quickassist
Signed-off-by: Haibin Huang<EMAIL_ADDRESS>

I see...
Actually, I think  zfile_compress_software() is a special case in  zfile_compress_qat() which the number of batches is only one..
So how about just changing the parameters of ICompressor::compress() as compress(const unsigned char *const raw_data[], size_t steplist[],  unsigned char *compressed_data[], size_t compressed_len[], ssize_t cur, size_t dst_len) instead of a new interface?

and check if QAT enable in compress() method itself

I see... Actually, I think zfile_compress_software() is a special case in zfile_compress_qat() which the number of batches is only one.. So how about just changing the parameters of ICompressor::compress() as compress(const unsigned char *const raw_data[], size_t steplist[], unsigned char *compressed_data[], size_t compressed_len[], ssize_t cur, size_t dst_len) instead of a new interface?

good idea, Yes, this is the easiest way, but you need to modify all the places where you call compress. I wonder if you can accept it

We only use the method 'ICompressor::compress()'  in zfile.cpp, so I think modifying the original parameters of 'compress()' is OK. Or maybe I can try to change some code in zfile.cpp so you only need to implement QAT in ICompressor?

We only use the method 'ICompressor::compress()' in zfile.cpp, so I think modifying the original parameters of 'compress()' is OK. Or maybe I can try to change some code in zfile.cpp so you only need to implement QAT in ICompressor?

It would be best if you could modify zFile to be compatible with batch commit processing, and it would be easier for us to modify.

and check if QAT enable in compress() method itself

ok

I create a PR in your repo.
BTW, please join our slack channel in CNCF
Login to the workspace through https://slack.cncf.io/  and search 'overlaybd'.
Thanks

I create a PR in your repo. BTW, please join our slack channel in CNCF Login to the workspace through https://slack.cncf.io/ and search 'overlaybd'. Thanks

ok, thank you, I have seen the PR.

@BigVan according to your comments and patch, we have modified it, please review it again.

LGTM


Using the rotation parameter of an emitter causes the position value to be multiplied by -1

If you set a rotation about some axis and some angle, instead of being located at position (x,y,z), the emitter position becomes (-x,-y,-z) and the rotation is set about that point instead. The expected behavior would be the emitter position stays at (x,y,z), and it rotates about that point (since the default center point is the position).

Note, the fix for this is super easy. Just multiply the end result of getRotation in rotationFunctions by -1, and the issue seems to be fixed.

Ha, just done that myself! I'll run a few tests and make sure it doesn't have any unexpected side effects :)

Thanks for fixing this stuff super quick!

No worries! The joys of having simple bugs combined with email notifications ;)

Multiplying by -1.0 inverts the value of the forces being applied (velocity/acceleration)...

Should be fixed now. Bumped to v1.0.1.


Use Culture.Parent for culture fallback

Addresses #14452 #14558

FYI, while I'm working on this I discover another bug that I already solved in Orchard Core, seems I forgot to file an issue before 3.0.0
I will try to reproduce and fix it in https://github.com/aspnet/extensions

@ryanbrandenburg why ResourceManager.GetString() fallback to the parent cultures by default? Is it a bug? coz this behavior will conflict with LocalizationOptions.FallBackToParentCultures

@ryanbrandenburg why ResourceManager.GetString() fallback to the parent cultures by default? Is it a bug? coz this behavior will conflict with LocalizationOptions.FallBackToParentCultures

If you've got a repro on an issue this causes file a separate bug.

If you've got a repro on an issue this causes file a separate bug.

I'm sure it's a bug, but shall I posted here or in CoreFx repo, while it's related to ResourceManager?

If the conflict is in LocalizationOptions.FallbackToParentCultures file it on us, even if that means we'll have to work with the CoreFx team to get a change.

The issue with the ResourceManager itself, I can write a unit test to prove that

There's been no update here in a week so I'm going to close this out. @hishamco, if you come up with a solution to the problem I stated above feel free to either re-open this or open a new PR.

Hi @ryanbrandenburg I was blocked by the ResourceManager bug that I mentioned earlier. I was waiting for your feedback :)
Could you please reopen it, coz all my unit tests did what it supposed to do, just one of them I didn't push it fail the test

@hishamco thanks for your effort. Do you plan to follow up on this PR or do you want to abandon it?

@mkArtakMSFT of course I will do, I was waiting for @ryanbrandenburg reply according my latest suggestion

@hishamco it was unclear to me that you were making a specific suggestion. What suggestion are you making?

I come up with another option is to define something like MaxCulturesLimit with the default value 5, so whatever the browser it sends it will only take the first 5 culture which is fine instead of all the known cultures

What exactly would MaxCulturesLimit control? Because if it truly limits the site to the first 5 cultures for which calls were made that won't work (having the site not support french one day then the next day after it's been restarted (unbeknownst to the users) having it support french is not good UX). If it caches the first 5 cultures then creates a new one every time after that, that won't work because it only adds a short extra step before the DDOS can begin.

What exactly would MaxCulturesLimit control?

Will control the number of the culture instances to be created at least

(having the site not support french one day then the next day after it's been restarted (unbeknownst to the users) having it support french is not good UX)

This is a problem if the current implementation itself :smile: I already logged an issue to support dynamic cultures without restarting the app

If it caches the first 5 cultures then creates a new one every time after that, that won't work because it only adds a short extra step before the DDOS can begin.

How it will creates a new one every time after that?!! they are already caches
The idea basically like what we need for HTTP language headers

What exactly would MaxCulturesLimit control?

Will control the number of the culture instances to be created at least

(having the site not support french one day then the next day after it's been restarted (unbeknownst to the users) having it support french is not good UX)

This is a problem if the current implementation itself :smile: I already logged an issue to support dynamic cultures without restarting the app

If it caches the first 5 cultures then creates a new one every time after that, that won't work because it only adds a short extra step before the DDOS can begin.

How it will creates a new one every time after that?!! they are already caches
The idea basically like what we need for HTTP language headers

Thanks for pushing for this, @hishamco.
After reviewing this internally we've decided not to move forward with this change. The reason is that this change is going to impact a lot of customers, yet we haven't heard much customer concerns regarding the current behavior.

No problem


temp diff to ask about model name

temp diff to ask about model name

Stack created with Sapling. Best reviewed with ReviewStack.

-> #847
#827
#826

Closing this and bringing out a proper fix


Fix create_proxy! macro

error: unexpected end of macro invocation
  --> /home/travis/build/denzp/rustc-llvm-proxy/target/debug/build/rustc-llvm-proxy-8a4ff3d82bd298fa/out/llvm_gen.rs:6:152
   |
6  | create_proxy!(LLVMContextSetDiagnosticHandler; (); C : LLVMContextRef , Handler : LLVMDiagnosticHandler , DiagnosticContext : * mut :: libc :: c_void ,);
   |                                                                                ^ missing tokens in macro arguments
   |
  ::: src/lib.rs:93:5
   |
93 |     macro_rules! create_proxy {
   |     ------------------------- when calling this macro

https://travis-ci.org/denzp/rustc-llvm-proxy/jobs/625547316
This patch fix it by removing , at the end

I'll try to revive #7 to pass tests on linux and osx and then publish to Crates.

Oh, scrap the last message: I've published v0.1.11 with the changes!

Also, the v0.2.0 is out - it uses libLLVM that can be found on Linux and MacOS and it's generally more reliable approach. Unfortunately, both v0.1 and v0.2 are broken on Windows.


Fix API inconsistence of ccui.Button's title renderer in web and native engine

https://github.com/cocos2d/cocos2d-js/commit/d810b38e2140d98b6aceb3033b225234021f016a
Web engine's enableStroke function in fact maps to enableOutline function in JSB.

The reason is ccui.Button in the native engine use cc.Label while cc.LabelTTF is used for the web engine

Fixed by #1578


Can not complete path after --src/--dst option in put/get subcommand

when use put/get subcommand in v0.6.0, can not auto complete file/dir path by TAB after --src/--dst option

fix at v0.6.1


Converts the whole file instead of selected area

Tabifies / untabifies the entire file even though only a portion is selected

Yes, that's the intended behavior. If you want to replace one with the other in a selected area, the find-and-replace package works well for that.