Unit tests & acceptance runs using Github Actions

I was just interested in Github Actions, and some project with complex requirements for testing. This is mostly based on work done here, and probably can fit into #3275. There are some issues with running tests on Windows Server platform: might be due misconfiguration of environment, or incorrect Python interpreter versions.
You can see the results of the CI run on this page.
More information about Github Actions can be found here.

Awesome work! Great to see Actions in action with rfwk :)

I have no previous experience with GitHub Actions but this looks great! Do I get it right that Actions allow all kind of automation on GitHub and one of the main usages is CI?
I looked at the CI results and list of provided OSes and Python interpreters is pretty awesome. It seems everything we support is covered. A related issue is that there are lot of failed CI runs, especially on Windows, and debugging/fixing them is not a small task. Is someone interested to look at them with me? On a quick look some look easy to fix and I think I know the root cause for Windows failures as well.
Some general questions about this CI solution:

Are results somehow shown on the project main page?

Can we have different jobs so that tests are run with some interpreters after every commit (preferably with a small grace period) and with others twice a day (if there are changes) or something like that?

Is it possible to force a test run without a commit?

Is there any way to get access to the generated logs and reports? Debugging failures when you only see the console output is not easy, especially if the problem only occurs on CI and not on your local machine.

There is a way to show a badge with a status on a page you want via editing Markdown file (e.g. README.md).
Example of badge for the workflow being discussed here:
The link to such picture looks like that: https://github.com/tysonite/robotframework/workflows/Unit%20&%20Acceptance%20tests/badge.svg
Some details in official docs.

Yes, this is possible to configure workflows as you wish: trigger jobs on events, schedule them, and much more. As well, you can have multiple separate workflows. See here and here. And basic concepts are defined here.

It is possible for the particular job that has been executed before. The top right corner if you select a particular workflow:

But I suspect it will run for the same commit as previously.

Any artifacts can be uploaded as a result of job. Just need to specify paths to files correctly in YAML workflow file.

Going back to the first question, I don't feel I would be able to support fixing of issues on various platforms due to my limited knowledge of Python. But I could defenetely support further changes to workflows, only need to specify requirements more specifically.

My preference would be that we land this PR as is before more changes are made.

@cclauss I feel that #3390 is a better option as acceptance tests run takes a long time to complete. Unit tests can be run on push / pull request events (only around 5-9 minutes to complete), and acceptance tests can be run on pull request / schedule events only.

@pekkaklarck, it would be nice to get more feedback from you. So far I shared 2 options as PRs:

Current PR (probably requires some additional tweaks in Github Actions workflow files similar to some fixes in #3390)
#3390 - another option, which looks better from my perspective.

this link points to already happened runs on 2nd option

Anything you would like to change from workflows perspective? If something is unclear, I will try to search for an answer.

RF + CI services like Github Action and CircleCI etc. give great possibility for parallel test execution. @tysonite You can get some inspiration from my CirclieCI pipeline for another open source project. The configuration in YAML format is very similar.
The idea is simply to split the execution (i.e. by test suites or tags) and have separate jobs for each part. At the end use rebot to merge the results output into one log/report.html
https://circleci.com/workflow-run/e1406d12-aeba-403e-b9f4-f38f31fcb9d0

config.yml --> https://github.com/ehrbase/ehrbase/blob/develop/.circleci/config.yml
log/report.html as artifacts --> https://circleci.com/gh/ehrbase/ehrbase/1688#artifacts/containers/0
@pekkaklarck  same feature is supported by Github Actions.

One could even push the result htmls to a separate repository and host this as Github Pages. And if one like to get even more creative one could combine that with Shiva's RF-Metrics library and host some really nice Dashboard as static Github Pages which would be automatically updated after each build on master branch :wink:
Another idea (that I already implemented for above project) is to push the rf-metrics report to a Slack channel :sunglasses:
This is how the Slack message looked (but that could also be static html hosted as Github Pages)

Just got another cool idea: the Slack solution is based on doing a screenshot with Selenium-Library in a headless browser in CI. Instead of sending that screenshot to Slack one could push it to RF's repository and have a image link reference in README.md (either in project root or in tests readme).

found my experiments on RF atests on CircleCI --> https://circleci.com/workflow-run/e88f8f30-2a39-40e6-9cba-e5bdaf0c5c1c
https://github.com/Tset-Noitamotua/robotframework/blob/feature/CircleCI-Pipeline/.circleci/config.yml
Basic idea of RFs atests run in parallel
robotframework_atests:
    jobs:
      - run unit test
      - run atest cli:
          requires:
            - run unit test
      - run atest core:
          requires:
            - run unit test
      - run atest external:
          requires:
            - run unit test
      - run atest keywords:
          requires:
            - run unit test
      - run atest output:
          requires:
            - run unit test
      - run atest rebot:
          requires:
            - run unit test
      - run atest rpa:
          requires:
            - run unit test
      - run atest running:
          requires:
            - run unit test
      - run atest standard libraries:
          requires:
            - run unit test
``

Quick comments and questions:

This looks awesome in general!
Unfortunately my knowledge about these solutions is pretty limited and so is the time I have available. It is thus impossible for me to evaluate different solutions like this PR, #3390, and Circle CI recommended above.
Luckily there are others in our local Robot Framework community (also members of Robot Framework Foundation) that know this stuff better and who can help with reviewing different options and setting up CI in general. I've already discussed with some of them and they promised to help.
I definitely want to get this implemented one way or the other. I'll submit a separate issue about setting up CI where we can continue discussion about the overall topic. I'll also summon up those "trusted souls" that can hopefully help drive this forward.
I'd be more than happy to help fixing tests on different environments. Tests that turn out to be hard/impossible to fix can be easily excluded by using tags.
Probably good to start with easy environments and start adding others one-by-one to avoid having lot of failing rungs.


Wrong security groups on machines created from nodepool

The default security group gets assigned to machines created from a nodepool instead of the expected
[parent-infra]-worker-sg.
This results in management cluster nodes not being able to communicate with the hosted cluster machines.
In addition to that, the current CAPI AWS provider doesn't honor a name filter for security groups. It expects that the ID is specified in the template, causing a panic if it's not:
https://github.com/openshift-hive/cluster-api-provider-aws/blob/e1fd3ca87e0b3445d17d5aab877d5b90046dfa5a/controllers/awsmachine_security_groups.go#L75-L78

/assign @enxebre


Failing workflows

Hi @AmeerTaweel, just to notify that workflows fail. I don't quite know the cause

Thanks for telling me.
The error that shows up is:
error: The source code is invalid, as it does not match the expected hash in the lock file.

So i updated the lock file of the auto-updater. Hopefully it will solve the problem. Will see in the next run.

Ok so in the last run one job succeeded and one failed due to a timeout. Now I have to solve the timeout issue...

Hi, @AmeerTaweel, fails again :(

I ditched using the the lockfile altogether. It seems the hash keeps changing regularly. Which is weird as I'm only using Deno's Standard Library. It seems they don't use versioning properly.
Regardless of the reason why this keeps happening. Not using a lockfile is not a great idea as well. It will let the workflows succeed for now, but I need to do something about it...

Hi! One problem was cross-platform extensions like this one. Apparently, nvfetcher wasn't ready for them - as shown in workflow logs.

Hi @deemp.
Sorry for my very late response. I was busy with my university semester.
I see that you created a PR, and then closed it and deleted the branch.
I also can see that the action was working on your fork until two weeks ago. It started to fail again since then.
Can you please explain the situation to me?

Hi @AmeerTaweel!
I want the extensions to be fetched in parallel and combined into .nix files. When this approach worked, the action takes just a couple of hours to complete.
The general approach is as follows. We first split the large  toml file into multiple files, and these files are processed in parallel in some number of blocks, so that a failure in one block doesn't affect other blocks. Then, we combine that number of .nix results for each block into a .nix file, and then combine these files into a single generated.nix.
To make this approach work, I used several Python scripts (see the scripts dir). Take, for example, VSCode Marketplace. The sequence is as follows.

Run updater to get nvfetch/vscode-marketplace.toml
Break it into blocks of some size (#extensions in each block), specify  and the number of blocks for each job. (scripts/generate-configs.py). You can run generate-configs.py: nix develop && nix run .#scripts.generateConfigs
In each job, run nvfetch.py that will fetch its given number of blocks and commit them into blocks/generated/vscode-marketplace/generated-<action-id>.nix. For each block, nvfetch.py runs nvfetcher. If nvfetcher fails, the extensions in the block are skipped. When nvfetcher processes a block, it produces the .nix files of its usual format (function arguments, function body, like in generated/vscode-marketplace/generated.nix). So, at the end of the loop, these files are combined (append_json, apend_nix) into a single file and this file is copied into blocks/generated/vscode-marketplace/generated-<action-id>.nix. To exclude race conditions during commits, the actions with lower  commit first.
Finally, scripts/combine.py combines all files in blocks/generated/vscode-marketplace/generated-<action-id>.nix and writes them into generated/vscode-marketplace/generated.nix.

Currently, we have a problem with ratelimits like this. Idk why this error is triggered on some jobs while other can continue.

Thanks for the explanation.
I first thought it was a rate limit from the marketplace itself, but it seems to be a GitHub thing.
Maybe updating all extensions everyday using GitHub actions is too much. If we only update a subset of them everyday it will still be fine. I don't think users care about getting all the updates on the same day. What do you think about this?

I guess the more extensions we have, the less frustrated will be the users of this flake.
As I have mentioned, usually, only one of the parallel jobs fails. Perhaps we can exploit this feature. We should spawn a separate job  in another workflow  that will check if at least one job in the main workflow has failed. If yes, the  will cancel the workflow and re-run the failed jobs. When the main workflow finishes, controller should finish too. I'll try to implement that

You're right, updating all of them is better of course.
And thanks for your help.

Hi there! I simplified the idea and just used the retry action. It worked today.

Amazing! I think it's time to merge it.

I asked if we are allowed to get extensions info in this way. Waiting for the answer. If you don't want to wait for an answer, let's merge!

Anyway, I made a PR #6


Add support fo *.ttinclude

At the moment only *.tt files are support.
Please also add *.ttinclude support for color highlighting.
Thank you!

That's a good idea, @iwhp :+1: Thanks for submitting it.
I think this only requires exporting an additional FileExtensionToContentTypeDefinition to associate .t4 extension with the TextTemplate content type in the TemplateContentType.cs.

any progress on this?

PRs are welcome 😄.


Match error of jsx tags

Please take a look at this tag match error.
The child div is self-closed, so it should not math the parent's close tag.

This is an atom issue. See atom/bracket-matcher#203


CI enhancements.

Add CI-legacy.yml workflow to build on older Xcode versions.
Update README.md to document availability of MoltenVK binaries via CI artifacts.
Update CI.yml workflow to use Xcode 12.4.

@past-due
I don't seem to be able to add you to the Reviewers request list. Don't know why.
Anyway...can you review as well, please?

I don't seem to be able to add you to the Reviewers request list. Don't know why.

Probably because he/she is not a member of KhronosGroup.

@billhollings Will take a full look. (From a first glance, there are a few ways this could be done without two separate workflows - unless you prefer having things duplicated / split like that?)

there are a few ways this could be done without two separate workflows - unless you prefer having things duplicated / split like that

I did think about trying to do this in one workflow (although I am a GH-CI newbie, so didn't look into that in much detail for now), and I'd definitely like to get your insights and suggestions on that.
As you can see from the second file, for legacy, we need to support a different matrix of platforms (which might evolve distinctly in the future), and artifacts should not be created. In addition, I'd like to keep the two workflows as identifiable as separate in the GitHub web UI, so we can enable and disable them separately via the web UI.
Essentially, what is meant by "legacy" in terms of platforms to build and versions of Xcode, will evolve over time. And at times, we may not even need a legacy workflow enabled.

Probably because he/she is not a member of KhronosGroup.

Hmmm. Possibly. Or maybe they need to be on the repo's official list of collaborators.
@past-due I've added you there now. You should get an invite.

@past-due I've added you there now. You should get an invite.

Accepted. (I believe it should now let you officially request a review.)

(I believe it should now let you officially request a review

Yup. There you are.

@past-due @cdavis5e Ping for approval.

@billhollings I've opened PR #1238 as an example of a different way this can be accomplished.

@past-due Thanks. #1238 is a much more elegant solution. I'm going to merge that PR and close this PR.


[Core-Apache HTTP Client] Implementation + Perf

Init

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-core


Fix bugs introduced in https://github.com/VirtualActuary/MiscVBAFunctions/pull/127

We need a better isDate() function.
After thinking about it we need a function to check if it is a string that Excel will auto convert to something else and then comment it out. To avoid any Excel auto-conversion.

See https://github.com/AutoActuary/autory/issues/1150


Fix RetrieveClassName Regex

RetrieveClassName Regex doesn't stop for the enclosing double quote if additional attributes follows:

If you change to . to [^\"] it will fix the issue:

Hi :)
Hmmmm, how did i test it..
Thanks, i will check it. Maybe i need namespsace as well..

Btw, could you please fix another case as well
x:Class=     "Test.ClassName" (spaces)
Or i will do it myself then :)

Sure, give me some time.

Better? :)

Yeah) like a boss

You might want to double check - is [:space:] working for you? Might be better off using \s instead. Sorry for that.

@Dresel hm, seems to be working, no?

Oh, this seems to be .NET Core specific - there it doesn't work.

https://github.com/AndreiMisiukevich/HotReload/blob/master/Reloader/Xamarin.Forms.HotReload.Reloader/HotReloader.cs#L810 would you like to send another PR with fix for both regex entries?

Sure, give me some time.


Call toJS() when serializing Immutable data structures

This PR add better serialization for Immutable JS data structures

Thanks for your contribution. let me work on a new patch release for jsx-to-string with this addition.

released 1.1.0 with support for this


Clean up and combine auto translations

This will be the most time-consuming task. There are two translations, one from Google Translate and one from Amazon Translate. Both are pretty rough, but between them there may be enough info to get one workable translation.

Closing this last issue as it's obvious.

Closing this last issue as it's obvious.


Allow passing of falsey option values

If any falsey value is passed as an option value it is
ignored instead of trying to add as a cli argument.
fixes #10
pr #11 previously fixed #13

Great, thank you @darrinholst

Released in v0.3.2


Having issue add panel with sankey-panel

Hi,
When i am trying to use the plugin , i get following error below , the plugin is under
/usr/share/grafana/public/app/plugins/panel/sankey-panel
Please adice
Thanks
Plugin Error
Fetch error: 404 Not Found Instantiating http://<IP_ADDRESS>:3000/public/app/plugins/panel/sankey-panel/module Loading app/plugins/panel/sankey-panel/module

Same here. Whereever i try to install it.

@talbaror and @int2001 have you mentioned the directory of sankey-panel plugin in grafana.ini
https://github.com/kumaravel29/sankey-panel#clone-into-a-directory-of-your-choice

@talbaror and @int2001 have you mentioned the directory of sankey-panel plugin in grafana.ini
https://github.com/kumaravel29/sankey-panel#clone-into-a-directory-of-your-choice

Hi
yes i did and restart grafana same result
Thanks


Add support for NMake project files

Our project currently defines the following nodes under the PropertyGroup nodes:
NMakeBuildCommandLine, NMakeReBuildCommandLine, NMakeCleanCommandLine, NMakeIncludeSearchPath, NMakePreprocessorDefinitions.
It seems like none of these are read (realistically, only the include path and preprocessor definitions are relevant) and passed on to clang-tidy.

Hi,
Thank your for your suggestion 👍
We'll look into it.

Can you provide an explanation of what these properties do?
It's not clear to me how Visual Studio processes them.

They act just like the regular Includes and Preprocessor keys, but aren't tied to ClCompile. They live under PropertyGroup.
You can find an example of their usage on this project: https://github.com/GameTechDev/ClusteredShadingAndroid/blob/master/ClusteredShadingAndroid.vcxproj#L39-L40

Hello,
We're still having trouble understanding this feature request.
Under a test project, based on your GameTechDev github sample, the NMAKE properties are fully loaded by clangpowertools.

VERBOSE: SET_VAR NMakeBuildCommandLine: "\VisualGDB.exe" /build "" "/solution:" "/config:Debug" "/platform:Win32"
VERBOSE: SET_VAR NMakeCleanCommandLine: "\VisualGDB.exe" /clean "" "/solution:" "/config:Debug" "/platform:Win32"
VERBOSE: SET_VAR NMakeReBuildCommandLine: "\VisualGDB.exe" /rebuild "" "/solution:" "/config:Debug" "/platform:Win32"
VERBOSE: SET_VAR NMakePreprocessorDefinitions: __VisualGDB_CFG_Debug;
VERBOSE: SET_VAR NMakeIncludeSearchPath:
C:\Android\ndk\platforms\android-18\arch-arm\usr\include;C:\Android\ndk\toolchains\arm-linux-androideabi-4.6\prebuilt\windows\lib\gcc\arm-linux-androideabi\4.6\include;C:\Android\ndk\toolchains\arm-linux-androideabi-4.6\prebuilt\windows\lib\gcc\arm-linux-androideabi\4.6\include-f
ixed;
VERBOSE: SET_VAR NMakeForcedIncludes: C:\test\win\gcc_Debug.h;\gcc_compat.h;

Should there be more to it than just loading them as regular project properties?

Hi,
I am on vacation from work where the projects live, but based on your output, it looks like these are being read now (possibly due to changes since the construction of the issue?). I think it is fine to close this issue for now and I'll re-open it if the problem is still present when I can check in a few weeks.


Fixes #677 - [coverity] double_free: Calling qd_free_http_adaptor_config frees pointer config which has already been freed

Codecov Report

Merging #683 (3012444) into main (69dcda7) will increase coverage by 0.93%.
The diff coverage is 87.67%.

@@            Coverage Diff             @@
##             main     #683      +/-   ##
==========================================
+ Coverage   25.90%   26.84%   +0.93%
==========================================
  Files         128      129       +1
  Lines       31189    31233      +44
  Branches     4972     4981       +9
==========================================
+ Hits         8081     8383     +302
+ Misses      22053    21768     -285
- Partials     1055     1082      +27

Flag
Coverage Δ

unittests
26.84% <87.67%> (+0.93%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/adaptors/http_common.c
17.36% <85.71%> (+17.36%)
:arrow_up:

tests/c_unittests/test_http_listener_connector.cpp
88.13% <88.13%> (ø)

src/policy.c
25.89% <0.00%> (-0.04%)
:arrow_down:

src/entity.c
40.55% <0.00%> (+0.41%)
:arrow_up:

src/adaptors/http1/http1_client.c
2.78% <0.00%> (+2.78%)
:arrow_up:

src/python_utils.c
28.57% <0.00%> (+3.57%)
:arrow_up:

src/vanflow.c
43.35% <0.00%> (+10.92%)
:arrow_up:

src/router_core/address_watch.c
88.50% <0.00%> (+14.94%)
:arrow_up:

src/error.c
45.05% <0.00%> (+15.58%)
:arrow_up:

... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Advance versions from 0.14 -> 1.0

Pending after this:

[ ] Test by pointing BoringSSL.podspec to latest commit of its repo.
[ ] Tag that commit of the BoringSSL repo.
[ ] Tag gRPC repo.
[ ] Build the plugin, zip it, and add it to the release.
[ ] Push Cocoapods to trunk.

Argh, of course this fails until we tag the BoringSSL repo...

templates/gRPC-Core.podspec.template may need to be updated too

Good catch!

Note that the 3 Travis failures are expected, as the apps under /examples use the released pods, and 1.0 isn't released yet. The apps under /src/objective-c/examples compile everything from sources, same as the tests, so they pass.

Since all the changes are about obj-c, and the tests are expected to fail because cocoapods doesn't know about 1.0.0-pre1, I'm merging by force.


Add building block to create custom precise assertions

Check List:

Unit tests : YES
Javadoc with a code example (on API only) : YES
PR meets the contributing guidelines

The goal is to be able to make precise soft assertions on complex objects (i.e. where we don't need to assert the equality of the full object) and to have easy to read test code and assertion error messages.
Example:
assertThat(customer).is(
    customer(
        name(
            first("Boris"),
            last("Johnson")),
            address(
                firstLine("10, Downing Street"),
                postcode("SW2A 2AA")
         )
    )
)

and on failure get an error message like:
[✗] customer:[
   [✗] name:[
      [✓] first: Boris,
      [✗] last: Johnson but was Johnstone
   ],
   [✗] address:[
      [✓] first line: 10, Downing Street,
      [✗] postcode: SW2A 2AA but was SW1A 2AA
   ]

I tried using custom soft assertions, but I found the error message to be less readable (and the implementation more complex).
Maybe there is already another way to achieve this?
Also NestableCondition is not a great name :relaxed:.
Looking forward to getting your feedback.
Thanks!

Thanks @alex859 , I will review your PR in the next couple weeks!

This assertion API looks lika Hamcrest, and it is known for its poor discoverability.
In other words, developer who writes customer( has no possibility to figure out the available continuations.
Have you considered using lambdas instead of nesting function results? Even though Java's nested lambdas are a bit verbose, they are discoverable.
E.g.
assertThat(customer).is(c -> {
    c.name.is(n -> {
        n.first.isEqualTo("Boris") // this allows to use full AssertJ power
        n.last("Jonson") // it might be helpful for literals
    })
})

With Kotlin, it would become
assertThat(customer).`is` { // "is" is reserved, so alternative is required
    name.`is` {
        first.isEqualTo("Boris") // this allows to use full AssertJ power
        last("Jonson") // it might be helpful for literals
    }
}

The key difference from the proposal in PR is that users get autocomplete after they type c.

Integrated thanks @alex859!

Thanks @joel-costigliola!

@vlsi Thanks for your comment! I'm not sure how the first Java example could be implemented, what type is c.name?
Kotlin devs can define a DSL to build nestable conditions that are discoverable.
Something like:
@DslMarker
annotation class CustomerDsl

@CustomerDsl
class CustomerConditions {
    private val conditions = mutableListOf<Condition<Customer>>()

    fun name(nameConditions: NameConditions.() -> Unit) {
        conditions += NameConditions().apply(nameConditions).build()
    }

    fun build(): Condition<Customer> =
        nestable("customer", *conditions.toTypedArray())
}

@CustomerDsl
class NameConditions {
    private val conditions = mutableListOf<Condition<Name>>()

    fun first(expected: String) {
        conditions += verboseCondition(
            { expected == it.first },
            "first: $expected",
            { " but was ${it.first}" }
        )
    }

    fun last(expected: String) {
        conditions += verboseCondition(
            { expected == it.last },
            "last: $expected",
            { " but was ${it.last}" }
        )
    }

    fun build(): Condition<Customer> =
        nestable("name", { it.name }, *conditions.toTypedArray())
}

And then use invoke as you suggested
operator fun ObjectAssert<Customer>.invoke(
    conditions: CustomerConditions.() -> Unit
): ObjectAssert<Customer> =
    `is`(CustomerConditions().apply(conditions).build())

to get:
(assertThat(customer)) {
    name {
        first("Boris")
        last("Unknown")
    }
}

Kotlin devs can define a DSL to build nestable conditions that are discoverable.

@alex859 , I guess the same (or even better DSL) can be created without resorting to Condition and Nestable.
Already existing assertSoftly is more-or-less fine for creating DSL like you suggested, and the implementation would not be limited by the small amount of existing Condition implementations.

@vlsi the main advantage of the proposed approach is that on failure the error message is more readable.

Technically speaking, "address" string should come together with function that extracts the address. Otherwise it is just wrong to link "address" and allOf.
There might be anyOf, noneOf, and so on, so labelling allOf with "address" is wrong.
If the functions are labelled properly, then the library can produce nice messages.


[release-4.13] Bug 2178033: Fixes issues with Alerts and Status not being propagated to Node level

Performing manual cherrypick due to GH issues.

/cherrypick release-4.13-compatibility

/lgtm


Set XDG_SESSION_TYPE earlier

Need pam_systemd module to see proper XDG_SESSION_TYPE setting, so need to export it before pam_open_session is called.

Hi, @roosemberth is already working on the same issue. Actually the freedesktop documentation indicates all XDG vars should be initialized before the pam session is opened. This resulted in Wayland not working in PR #137. It has to do with issue #165 as well, so there is a lot going on here...
Interestingly everything is fine under Wayland with this PR, but the variable is never set under X and tty (at least on my system). We will have to figure it out entirely before I merge :)

On Mon, Feb 3, 2020, 12:12 AM cylgom<EMAIL_ADDRESS>wrote:

Interestingly everything is fine under Wayland with this PR, but the
variable is never set under X and tty (at least on my system). We will have
to figure it out entirely before I merge :)

Weird. Sorry, I didn't think to test on X and tty before submitting, my
motivation was to get pam_systemd.so to set type=wayland properly.
Thanks for your work on this!

I think I could inspire myself from this PR to fix #137 (Sorry for the delay). I'll hopefully work on it in the upcoming days.


some of oneDNN and oneCCL samples changes their names

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant. Add a 'X' to the one that is applicable.

[ X] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[X ] Command Line

Checklist for Moving samples:
Links and Details can be found in the samples WG Teams Files.

[ X] Implement fixes for ONSAM Jiras

@louie-tsai  - can you look at the ci issues please

@louie-tsai - can you look at the ci issues please

Known errors due to incompatibility between Beta9 toolkits and Beta10 toolkit


saga1.5.2 重试机制在哪配置?我在状态图里面配了Retry,但是没效果

[ ] I have searched the issues of this repository and believe that this is not a duplicate.

Ⅰ. Issue Description
Ⅱ. Describe what happened
If there is an exception, please attach the exception trace:
Just paste your stack trace here!

Ⅲ. Describe what you expected to happen
Ⅳ. How to reproduce it (as minimally and precisely as possible)

xxx
xxx
xxx

Ⅴ. Anything else we need to know?
Ⅵ. Environment:

JDK version :
Seata version:
OS :
Others:

https://github.com/seata/seata/issues/5077


shaaleissa/basic-authentication-challange completion

@shaaleissa thanks for completing the challenge. You still need to do lint fixes to pass all the test cases.
In addition, if you notice there is a .pre-commit-config.yaml file which will help you to run health check against your code prior pushing to remote.
Pre-commit

installation: you need to install pre-commit

# ensure pre-commit hooks is installed
pre-commit --version

# execute pre-commit hooks against your code, make sure you are executing the command in the root of the file
pre-commit run

Also, you might execute flutter analyze in your flutter root project to ensure it will not fail due to lint rules.

@shaaleissa Please try to test it locally and make sure flutter analyze passes prior pushing to remote.

I cannot continue review your code as you still have major issues. Please fix it then push the latest changes.


[New Feature] Caption for images in ELN

Context:
As an academic/researcher using the ResearchHub Notebook, I would like to be able to add captions to images that I upload in order to better organize and contextualize my academic work. This feature would enable users to include a brief description or label for images uploaded to their Notebook, making it easier to identify and reference specific images when reviewing their work.
Requirements:

By hovering over an uploaded image, the user should see an icon to "Add Caption."
Users should be able to enter a caption of up to 150 characters.
The caption should be displayed below the image in the Notebook.
If a user adds a caption to an existing image, the caption should be added to the image and displayed in the Notebook.

References:
Notion has a solid implementation of this feature, which can be found here:
https://www.notion.so/help/images-files-and-media

Hi @andrewpedron, may I take this on. I've applied to both tasks on Dework but happy to get started with this one. I've introduced myself on Discord as well.


Feature/performance enhancement: Message Caching

This pull request implements message caching local to each process.  Messages are assigned unique message IDs and assigned to the local message cache.  Subscribers always check the cache for the message before trying to reconstitute a version of the message from shared memory.
This enhancement makes it such that no (de)serialization or message copies are required for pubs/subs that exist together in the same process.

In running test_perf against main, I am noticing a decrease in max effective send rate ~7ksamps/s vs ~9ksamps/s. Overall bandwidth appears to be slightly decreased as well. Running on x86_64-darwin, python 3.11.

@pperanich That's on me...  I had to change the message size of the perf test to debug something and never changed it back.  I've pushed a change that allows an apples-to-apples comparison.

It also turns out that calculating backpressure with every message was resulting in a pretty severe performance regression.  That's been addressed and I'm now seeing identical performance for small message sizes between the main branch and this one.


Traceback (most recent call last):   File "/home/miop/belajar/turbosnake/main.py", line 2, in      from turbosnake.ttk import tk_app, tk_button, tk_label, style, tk_frame, tk_grid ImportError: cannot import name 'tk_grid' from 'turbosnake.ttk' (/usr/local/lib/python3.10/dist-packages/turbosnake/ttk/init.py)

how to use tk_grid . try to provide any type of layout that is supported in this framework.

There is no such thing as tk_grid. There is a component inserter function named tk_grid_frame that creates a frame with grid layout. Here is an example of it's usage.

is there a way to make component material design with tkmaterial.

is there a way to make component material design with tkmaterial.

What is tkmaterial? If it's a theme for tk/tkinter then it probably will work with turbosnake.


[Snyk] Security upgrade com.amazonaws:aws-java-sdk-s3 from 1.12.223 to 1.12.338

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to fix one or more vulnerable packages in the `maven` dependencies of this project.
:sparkles: Snyk has automatically assigned this pull request, set who gets assigned.
As this is a private repository, Snyk-bot does not have access. Therefore, this PR has been created automatically, but appears to have been created by a real user.
Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

uasdm-fargate-erossync/pom.xml

Vulnerabilities that will be fixed
With an upgrade:

Severity
Priority Score (*)
Issue
Upgrade
Breaking Change
Exploit Maturity

616/1000  Why? Proof of Concept exploit, Has a fix available, CVSS 5.9
Denial of Service (DoS) SNYK-JAVA-COMFASTERXMLJACKSONCORE-3038424
com.amazonaws:aws-java-sdk-s3:  1.12.223 -> 1.12.338
No
Proof of Concept

(*) Note that the real score may have changed since the PR was raised.
Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
👩💻 Set who automatically gets assigned
🛠 Adjust project settings
📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Denial of Service (DoS)

Out of date


[Snyk] Security upgrade golang from 1.22.4-alpine3.20 to 1.23rc2-alpine3.20

This PR was automatically created by Snyk using the credentials of a real user.![snyk-top-banner](https://github.com/andygongea/OWASP-Benchmark/assets/818805/c518c423-16fe-447e-b67f-ad5a49b5d123)
Snyk has created this PR to fix 1 vulnerabilities in the dockerfile dependencies of this project.
Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Snyk changed the following file(s):

tool/actions-gh-release/Dockerfile

We recommend upgrading to golang:1.23rc2-alpine3.20, as this image has only 0 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Vulnerabilities that will be fixed with an upgrade:

Issue
Score

CVE-2024-5535 SNYK-ALPINE320-OPENSSL-7413532
  364

CVE-2024-5535 SNYK-ALPINE320-OPENSSL-7413532
  364

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

Ignore rc wait for the stable release version


Navbar collapse

I don't see any documentation on getting the collapsable navbar to work. Clicking the hamburger icon doesn't seem to open up the collapsed menu. Is there a component needed to get that to work?

The collapsable thing requires the collapse jQuery plugin in plain bootstrap.js, which is currently not available as an ember component, but will be soon!

bs-collapse component is availably as of now in ember-bootstrap v0.4.0! :)


[BUG] flaky test in index management dashboard plugin

What is the bug?
Rollup and Transform cypress tests are flaky and are failing in multiple linux and windows distributions.
https://github.com/opensearch-project/opensearch-dashboards-functional-test/blob/main/cypress/integration/plugins/index-management-dashboards-plugin/transforms_spec.js#L235
https://github.com/opensearch-project/opensearch-dashboards-functional-test/blob/main/cypress/integration/plugins/index-management-dashboards-plugin/rollups_spec.js#L249
Many a times newly created rollups and transform are coming as disabled instead of enabled.
How can one reproduce the bug?
https://github.com/opensearch-project/index-management-dashboards-plugin/issues/1095
What is the expected behavior?
Integration test should pass in release process
What is your host/environment?
linux rpm, linux deb, linux tar
Do you have any screenshots?
https://github.com/user-attachments/assets/48adda01-1662-4b44-80c8-3db5ed52aa59
https://github.com/user-attachments/assets/e2fe1906-3bda-46d8-ab48-9862a7b377f1
Do you have any additional context?
Add any other context about the problem.

There is no actual issue in 2.16 for linux:x64 or any other linux distributions. As in multiple runs this test fails sometimes.


add background color and dot color

Thanks for opening this PR!
A deployment should be available at https://18.qrcode.yaoss-collection.com/ shortly.
Have a great day!


COMP: Unused variable warning

Modules/Core/Transform/test/itkComposeScaleSkewVersor3DTransformTest.cxx:560:37: warning: unused variable 'pntT' [-Wunused-variable]
TransformType::InputPointType pntT = idT->TransformPoint(pnt);
^

@thewtex @dzenanz This is, hopefully, very safe change.  I propose that if the standard Linux/mac/windows build succeed, it gets pulled in quickly today. It would be nice if other PRs don't have this warning as a nuisance.

It would be nice if other PRs don't have this warning as a nuisance.

I fully agree. I was thinking about fixing this myself.


Removed commas.

There is a comma in the original English text, so the CSV separator is wrong.
Tentatively fixed for English and Japanese.
The sentences are broken, so the translation has been changed.
I think there will be a response to CrowdIn, so you can cancel the pull request if there is a problem.

Thanks for the reply.
I've added the quotes as you pointed out.
Since it is the original text, I tried to make the changes as small as possible.


Prepare release 0.4.1-alpha1

This is a pre-release to fix withdraw issues we currently have on chaosnet only. If this pre-release goes well, another "official" release v0.4.1` will be follow shortly today.

Close in favour of #1792


What Reddit user data to pull?

Hi, I'm trying out redditwarp, a Python wrapper for the Reddit API.
What data do we need to pull from Reddit? I'm assuming it has something to do with posts, comments, upvotes, downvotes. Is my assumption correct?

Hello. Yes. At the very least, for the Reddit user (repo user), we need to answer the question what subreddits, posts and comments am I reading, interacting with, and creating? Reddit chats and private messages are good too if they're available. The timestamp is important because eventually we'll categorize Reddit content consumed/created correlate the with other well-being metrics such as mood levels.


the swiper height is not correctly calculated, mobile orientation change

After page load, the swiper height is correct initially.
When resizing window to smaller, the the swiper height is re-calculated correct.
But When resizing window to bigger, the the swiper height is not correctly calculated most of time,
and images are partially hidden.
By most of time, I mean at certain window size, the height is correct.
On mobile Chrome and Safari, when changing orientation,  images are partially hidden. Only after swiping content up and down, the swiper height becomes correct and images are fully visible.
swiper-container and Images width is 100% in order to be responsive.
Thanks for help.
----   code -----

Slide with HTML
You can put any HTML inside of slide with any layout, not only images, even another Swiper!

Some content goes here. ....

HTML code is not correctly displayed. Is there a way to format HTML code? Thanks.


StaticMapExclusionFilterFactory and StaticListExclusionFilterFactory will not work via AccessControlSettingOperation.factory

ComplexAccessPoint.xml lists an example of how to apply an exclusion access control to requests made from outside a specified IP range:  https://github.com/iipc/openwayback/blob/master/wayback-webapp/src/main/webapp/WEB-INF/ComplexAccessPoint.xml#L65-88
Historically, we have used something very similar at the Library of Congress, but instead of using OracleExclusionFilterFactory, we use StaticMapExclusionFilterFactory, and follow the exact format outlined in the example that I linked above.  So, our code might look like:
<property name="authentication">
    <bean class="org.archive.wayback.authenticationcontrol.AccessControlSettingOperation">
        <property name="operator">
            <bean class="org.archive.wayback.util.operator.NotBooleanOperator">
                <property name="operand">
                    <bean class="org.archive.wayback.authenticationcontrol.IPMatchesBooleanOperator">
                        <property name="allowedRanges">
                            <list>
                                <value><IP_ADDRESS>/24</value>
                           </list>
                       </property>
                    </bean>
                </property>
            </bean>
        </property>
        <property name="factory">
            <bean class="org.archive.wayback.accesscontrol.staticmap.StaticMapExclusionFilterFactory" init-method="init">
                <property name="file" value="${wayback.basedir}/exclude.txt" />
                <property name="checkInterval" value="600" />
            </bean>
        </property>
    </bean>
</property>

This works for us in Wayback 1.6 and 1.8.  However, on the Openwayback 2.1.0 instance that we just installed, this same logic does not work.  Using the StaticMapExclusionFilterFactory as the exclusionFactory property of the AccessPoint itself works, something like:
<bean name="8080:accesspoint" class="org.archive.wayback.webapp.AccessPoint">
    ...
    <property name="exclusionFactory">
        <bean class="org.archive.wayback.accesscontrol.staticmap.StaticListExclusionFilterFactory" init-method="init">
            <property name="file" value="${wayback.basedir}/exclude.txt" />
            <property name="checkInterval" value="600" />
        </bean>
    </property>
</bean>

Likewise, using the StaticListExclusionFilterFactory as the AccessPoint's exclusionFactory property works.  Tomcat is able to restart without errors when using StaticMapExclusionFilterFactory / StaticListExclusionFilterFactory with AccessControllSetting.factory, but the application does not exclude URLs provided in the defined exclude.txt file.
I've reviewed some of the recent-ish commits to the code involved in this process, and nothing is jumping out to me as an immediate red flag.  After that, and doing loads of testing to no avail, I'm left scratching my head on this one.  I could definitely be overlooking something more obvious to the folks familiar with this code base, so if that is the case, please let me know.
I am not seeing any helpful information in my Tomcat logs, but if I can provide any further environmental/testing info to assist in the investigation of this issue, just let me know.
Thanks in advance for the assistance.
Phil /<EMAIL_ADDRESS>EDIT: I believe Gina brought up this same issue in 2014 in the following google group exhange:  https://groups.google.com/forum/#!topic/openwayback-dev/2nANsg14QJE

Do we happen to know if the example in the docs still works? I.e., do we know this is specific to StaticMapExclusionFilterFactory?

Are you referring to the following example in the ComplexAccessPoint.xml?  I have not tested it.  That is a database driven functionality, no?  That was my assumption, which is why I didn't test, as we don;t have the database backend.  If not, I could test it out, but am not sure what else would need to be configured for the Oracle features to work.
Also, we do know that this is not specific to StaticMapExclusionFilterFactory, as it is also an issue with StaticListExclusionFilterFactory.

I was. I haven't gotten into it very far, but I suspect the problem is most likely to be in IPMatchesBooleanOperator just because that's where the complexity is -- I'm guessing it never gets to the factory, in which case it would also be a problem with the OracleExclusionFilterFactory configuration. There haven't been a lot of changes in IPMatchesBooleanOperator betwen 1.6 and now, as far as I can tell, but tomorrow I'll write some unit tests and see what I can figure out.

Awesome, thanks.  I will give IPMatchesBooleanOperator another look as well.

I should note, the IPMatchedBooleanOperator itself works for me when applied to the authentication property without the exclusion list/bean.  For example, IP exclusion with the following access point property works as expected:
<property name="authentication">
      <bean class="org.archive.wayback.authenticationcontrol.IPMatchesBooleanOperator">
        <property name="allowedRanges">
          <list>
            <value>(IP RANGE)</value>
          </list>
        </property>
      </bean>
    </property>

This will request anyone from outside of (IP RANGE) to authenticate to access the AccessPoint.  However, the following does not work to achieve the opposite effect (require anyone on IP RANGE to authenticate and allow anyone outside of IP RANGE to access freely):
<property name="authentication">
      <bean class="org.archive.wayback.authenticationcontrol.AccessControlSettingOperation">
        <property name="operator">
          <bean class="org.archive.wayback.util.operator.NotBooleanOperator">
            <property name="operand">
              <bean class="org.archive.wayback.authenticationcontrol.IPMatchesBooleanOperator">
                <property name="allowedRanges">
                  <list>
                    <value>(IP RANGE)</value>
                  </list>
                </property>
              </bean>
            </property>
          </bean>
        </property>
      </bean>
    </property>

When I use the above, I am still able to access the AccessPoint from within IP RANGE.  When I am outside of that range, however, I no longer get a request to authenticate, but rather get a blank screen.
So, at this point, I am not sure if the issue lies in:

applying the StaticListExclusionFilterFactory factory based on the result of IPMatchBooleanOperator
the use of   AccessControlSettingOperation + NotBooleanOperator
both of these things

AccessControlSettingOperation always returns true, regardless of the operators -- what the operators do is determine whether it sets a filter on the request. So that makes sense, though it took me a while to figure out. :)

Let me make sure I understand the use case:

If someone's in the specified IP address range (e.g. <IP_ADDRESS>/24), return all results unconditionally.
If someone's outside that address range, return only those results whose URIs are not listed in exclude.txt.

Is that correct?

Yup, that's it exactly.  I was coming to the same conclusions as you re: AccessControlSettingOperation.  As far as I can tell:

the operator is indeed set
the operator logic works of its own accord (outside of this use case)
the factory is indeed set
the factory works of it's own accord (outside of this use case)

This, I am not sure why this wouldn't kick in when I'm off the IP RANGE and limit my access.

So, what we don't have is a proven case of ACSO + filter factory working (with any filter factory).
Here's my current theory: I have unit tests that show that the ACSO + factory setup should work, in isolation, but it's not working in the real world. So maybe tthere's something different between the code path when you set the exclusionFactory property directly on the AccessPoint, and the code path when you set it only indirectly via the ACSO. Either the filters don't get applied or, once applied, they don't get called, for some reason?
But it's going to take some digging into AccessPoint to examine that theory, and it's a monster, as AccessPointTest points out:
 * TODO: this unit test is too complex. it is because AccessPoint class has too
 * much responsibility and many execution paths. some good refactoring of
 * AccessPoint class would help.

@dmolesUC3, that seems like a reasonable enough theory, and AccessPoint seems like a logical place to check next, particularly because:

the logic seems to be there in the other pieces (ACSO, IPMBO, SLEFF, etc)
testing indicates that this functionality worked in wayback 1.8 and 1.6
there haven't been a whole lot of significant changes to the other pieces, while there have been significant changes to AccessPoint over time

I will continue to poke around.

As is, AccessControlSettingOperation sets the exclusionFilter on a WebRequest directly. In AccessPoint, it used to be that exclusionFilter would only be set on the WebRequest instance if the AccessPoint.exclusionFactory was not null. When that logic in AccessPoint moved into createExclusionFilter() between OpenWayback 2.0.0 and 2.1.0 (https://github.com/iipc/openwayback/commit/1a350bb2b57e47e1ceda655020a58f7e3c036a5a#diff-e00aad5f52c8452f5e58d3edc6a45479R292) the code changed so the exclusionFilter would always be set by AccessPoint, and if the exclusionFactory is null (like in LOC use case where the exclusionFactory is on AccessControlSettingOperation not the AccessPoint), the exclusionFilter is set to null.
I am proposing a fix that just has AccessPoint check if the WebRequest instance already has exclusionFilter set (not to null) to determine if it should call createExclusionFilter(). This way, if AccessControlSettingOperation sets an exclusionFilter, it will not get overwritten.
@ptrourke

Makes sense to me. Thanks for looking into this, Lauren.

Ouch, clearly my oversight. Now I fully understand the use case of authentication.
As the intent of 1a350bb was to (eventually) eliminate this kind of confusing way of configuring access-control / exclusion, will you let me propose a slightly different change + test cases...?
(if both authentication and `exclusionFilterFactoryis combined with

OK, I tried and found it takes too much effort to clean it up. I just added a test case to detect this issue.
Toward 3.0, I'd propose redesigning AccessControlSettingOperation as ContextExclusionFilterFactory and have it set up through AccessPoint.exclusionFactory. It'll require adding WaybackRequest (or narrower interface) to its 'getExclusionFiltler' method.
Until then, exclusion filter set up through authentication does not work with CDXServer  (sorry, I broke it in 1a350bb.)

I agree that AccessControlSettingOperation should not remain as it is and redesigning it using ContextExclusionFilterFactory makes sense.

Is there an issue open for such a redesign? If not, could someone please write it up.
Also, is everyone happy with the fix @ldko proposed in #287 and @kngenie addition in #289 ? If no one objects to those, I'll merge them tomorrow and close this issue.

This is our ticket, I don't think we have any objections, Lauren, do we (LC) need to write up the issue for the redesign?

I opened issue #290. It should be put on the 3.0.0 Major Release milestone.

Yes, thanks, @ldko, for looking at this!


Update strategy map

https://www.draw.io/#Uhttps%3A%2F%2Fraw.githubusercontent.com%2Fsara-sabr%2FITStrategy%2Fstratmap2%2Fdrawio%2Fen%2FStrategySummary.drawio

Added links to DORA capabilities.
The only one missing is Shifting left on security.  Its kind on included in Continuous integration (automated testing).


How can I get the embedded subtitles of the videos ?

I want to get the hardcoded subtitles from the videos (if it was exist) as an array instead of displaying it as a captions .. any help ?

@omar20261 Have you found a solution?
As I understand, right now this lib suggests using web-hosted subtitles which is totally inappropriate.

did you guys found the solution?


Need ability to flash with extra heat capacity (e.g. from catalyst or reactor mass)

FlashVL supports flashing to a given H.  But often (for example flashing after adding ∆Hrxn), one needs to add some additional heat capacity, for example from the catalyst or reactor mass.
But there's no mechanism to add extra substances that don't participate VLE (e.g. solids in thermal equilibrium).
A partial workaround is to multiply the ∆H by cp0 / (cp0 + extra_cp), where cp0 = current_eq.dH_dT_P().  But this is inaccurate if dH_dT_P() changes between the initial and final H.
Is it possible to flash with such extra substances in the existing API?  If not, could you add it?

I don't plan to do this work, so I am closing this issue.


Guided Navigation

Purpose: Use extension hook mechanism to support guided navigation, forward and backward, through an ordered list of items. Example use cases:

Using QnABot for a guided virtual tour or facilities or artifacts, user can say “next room please”, or “next item please”, and the extension function will redirect QnABot to answer the next question in the ordered list. Or “go back”, “Previous item” item to navigate backward in the list.
Using QnABot for training curricula, user can say “next topic please”, and the extension function will redirect QnABot to answer the next question in the ordered list of topic questions.

Implementation:

Automatically derive default question order from list of items sorted by ID.
Provide easy mechanism for user to supply custom ordered list of questions
a.	One way this could be done it to leverage the existing content designer to specifiy the ordered list of questions in a well-known QnA item ID, eg: ‘RESERVED.QnA.Question.Order” that can be queried by the navigation function to retrieve the ordered list of questions..  Other methods are also possible, but may require extra implementation effort.

If we allowed multiple topics per question you could implement this using overlapping topics

qid
topic_1
topic_2

1
A
-

2
B
A

3
C
B

and so on

fixed in 2.1


an error of preinstall.sh

an error of file extension
change "set-host-daemon-port-range.yaml" to "set-host-daemon-port-range.yml"

Coverage remained the same at 34.564% when pulling 86318464df7edc42675746756675e4d7657bf0c2 on guningbo:master into 5fc32be1b00bc89e7d019bbc535254d8589e58eb on microsoft:master.


Go Live [Dec 26]

Push to live

Learn Build status updates of commit 29cb8db:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/sbs-gs-nodejs.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


supersede PR 270 (identical except for signed commit)

Signed-off-by: Kaliya - Identity Woman<EMAIL_ADDRESS>

Kaliya expressed her sign-off intent in the comment stream on her other PRs, so I cherry-picked her commit and am merging it here.


fix obtaining LDAP schema

LDAP post schema is different than scheme for scope, needs to be resolved

fixed in commit https://github.com/JanssenProject/jans-cli/commit/08c6d50d17a1ba866ec068f4bd75a12009d928e9


where to ask questions about Cactus

I see that this issue tracker is often used as a forum to ask questions. I wonder if the developers are ok with this or would prefer another way.
I find Cactus very interesting but I'm a bit afraid about the lack of some kind of community support for users and developers. I'm aware that it's a small project, but I think that this step is needed to make it grow.
The easiest way would be probably using a specific tag to follow on stackoverflow. The cactus tag is already taken for Jakarta Cactus, now retired.
What would you recommend? cactus-python? cactus-web?
What do you think?

could be nice, but the devs might want all this discussion in one place so we'll have to see what they say.
another option, if the discussion grows a lot might be a google group.


PHPDoc changes on ReaderInterface and all of it's implementations

The ReaderInterface::fromString() method defines bool as a potential return value to allow reader implementations to return a boolean false when unable to process the configuration. In reality, most will raise exceptions. However, the implementations MUST conform to the interface, and changing the interface is a BC break, regardless of whether or not it is enforceable. As such, I'm closing this without merging at this time.


RSSOD-BENCH release time

Where can we download the RSSOD-BENCH dataset?

Hi, sorry for the late replay.
The dataset is available at huggingface.


[FLINK-27834] Copy all files except those filtered by .dockerignore

Signed-off-by: tison<EMAIL_ADDRESS>

cc @morhidi @SteNicholas

Thanks for your reviews!


Add kubetos

kubetos is a TOSCA framework to deploy Kubernetes developed by @Shishqa and available at https://github.com/Shishqa/kubetos.

Thank you for mentioning!
There is also an orchestrator clouni developed by ISPRAS, we would be happy for you to have a look at it too

@Shishqa you are welcome to open an issue for clouni.
Please provide me information describing clouni such as

TOSCA definition versions supported
Target environments, e.g. any cloud or a specific provider
Usage, e.g. CLI, Webapp, SaaS, etc.
Nature, e.g. open source vs commercial, academia vs company, etc.
Implementation languages, e.g. Python, Go, Java, etc.
Links, e.g. website, github, others
Is clouni using other TOSCA implementations? e.g. OpenStack TOSCA Parser, Puccini, xOpera, etc.
Any other key information

I will be happy to include clouni in the TOSCA implementation landscape.


[WIP] Automatically configure SSH key for user

This needs some testing still, but the idea is:
Can we make a fully automated flow, so that once somebody has a GitHub account, we can set up push access? (everything is opt-in)
This means, if a push fails due to invalid credentials:

we prompt them to ask if we should set up SSH access
if they don't have an .ssh key, we create one
we upload their public key to their github account (which requires write:public_key)
we verify the identity of github.com using a compiled-in SSH fingerprint

I couldn't find a GraphQL mutation for adding a public_key, so I use the API v3 here... but maybe I just missed it?
I thought it was going to be hacky but it's actually not too bad IMO. There's still a bug after github repo creation where it doesn't move to the next part of the flow automatically

That's really cool 💯


youtube-music: 3.6.2 -> 3.7.1

Changelog: https://github.com/th-ch/youtube-music/releases/tag/v3.7.0
Changelog: https://github.com/th-ch/youtube-music/releases/tag/v3.7.1

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

version 3.7.0 changed the WM_CLASS (app ID on wayland)  to com.github.th_ch.youtube_music
PR: https://github.com/th-ch/youtube-music/pull/2758
Issue: https://github.com/th-ch/youtube-music/issues/2687
We should update the StartupWMClass in the desktop entry to match that
- startupWMClass = "YouTube Music";
+ startupWMClass = "com.github.th_ch.youtube_music";

Just like the package in the AUR: https://aur.archlinux.org/packages/youtube-music

I don't know if other changes are needed to make it work on Plasma/GNOME, like changing the .desktop and/or icon names. See this: https://github.com/th-ch/youtube-music/issues/2806

@h-banii could you create a PR for that, please?


feat/wrapper

This PR includes:

The wrapper view
The wrapper view model
The wrapper view named route setup
Stacked basic setup

LGTM!


Discord Bridge

I'm hosting a service that connects channels in Discord, Mattermost, Slack, & Gitter.
Each connection can be unidirectional or bidirectional.
Current Bridges:
Discord #mattermost ⇄ Mattermost ~discord-bridge
Discord #rholang    ⇄ https://gitter.im/rchain/Rholang

(testing)
Discord #playground ⇄ Slack #Community

Goals:

Encourage Coop membership.
Utilize the strengths of each platform
Comply with Discord TOS (unclear for bots like this)

RHOC bounty - I have an unsigned SOW pending with the Coop. If accepted, rewards should be distributed to others, especially the dev of the bridge software.

Great !!
For the SoW and Issue rewards.
My idea for the SoW rewards is that the amount of SoW is stable and that
the issues your working on are subtracted for their part. So if your SoW is
for example 100,000 per month then your invoice would be something like
this:
SoW Bus.Admin 70,000
Issue 40       5,000
Issue 89      15,000
Issue 94      10,000
------+
Total        100,000
The SoW amount won't be public.
Finally the SoW becomes 0
My goal is to have all work being done, listed as an issue in GitHub.
My feeling at the moment is that the Board initiates issues without much
transparency.
--
Cheers,
HJ
On Wed, Sep 13, 2017 at 9:52 PM, drbloom<EMAIL_ADDRESS>wrote:

I'm hosting a service that connects channels in Discord, Mattermost,
Slack, & Gitter.
Each connection can be unidirectional or bidirectional.
Current Bridges:
Discord #mattermost ⇄ Mattermost ~discord-bridge
Discord #rholang    ⇄ https://gitter.im/rchain/Rholang
(testing)
Discord #playground ⇄ Slack #Community
Goals:

Encourage Coop membership.
Utilize the strengths of each platform
Comply with Discord TOS (unclear for bots like this)

RHOC bounty - I have an unsigned SOW pending with the Coop. If accepted,
rewards should be distributed to others, especially the dev of the bridge
software.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rchain/Members/issues/99, or mute the thread
https://github.com/notifications/unsubscribe-auth/AB0x9wZE1027glSzCQ2nfEG4uB1829GUks5siDJ0gaJpZM4PWorX
.

This is a link to the Discord Bridge channel in Mattermost


Understand and define sensible IP limits policies for Discv5

We disabled the max duplicate per-bucket and per-table limits for discv5 for the EthCC workshop. We don't understand the full implications of doing this. This needs to be looked into, and properly addressed.

One simple way out would be having a switch that disables this so we can run it in workshops (unsafe), and by default enforce the limits that are there.


Add exceeds_available_stock of Japanese translation

I do not know the reason why line_items is needed to translate a quantity.But if it does not exist, it has never been translated.

@project-eutopia Please review this.

@shioyama Please review this.

@shioyama ping


Bumping prometheus server resource limits for pangeo-hubs

pangeo-hubs prometheus server was in a OOMKilled/CrashLoopBackOff state again, so this PR bumps the resource limits for the server. This has already been deployed to the cluster and the server is running again.

Thank you @yuvipanda and @consideRatio!


fix: merge layers

:tada: This PR is included in version 1.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Node Click

Hi
Is there a way to implement node click.
If a node is already selected clicking the node again does not fire the activate event.  I'm using the treeview in a side bar for selecting a user and opening a module in the main page.  If I navigate from that module, clicking the treeview to open the same user again doesn't work as it is already selected.
Thanks
Nick

yes - use actionMapping


Error when using .netcore 3.0 and c# 8

I am trying to use the .netcore 3.0 preview and obfuscate my dll, it met this error:
Note that Rollbar API is enabled by default to collect crashes. If you want to opt out, please run with -s switch
  Loading project Obfuscar.xml...Processing assembly: MyProject, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=8d84e04138540ce2
  Loading assemblies...Extra framework folders:
  An error occurred during processing:
  Unable to resolve dependency:  System.Runtime.Loader

It work fine when I use the .netcore 2.x. It seems some relative with this issue:
https://github.com/dotnet/corefx/issues/22142

.NET Core 3.0 is far from complete at this moment. You will have to wait till it reaches RC.

FYI, dotnet core 3.0 RC1 is released!

Due to the bugs I personally identified in .NET Core 3.0 RC1, I don't really recommend you upgrade to it.
Besides, the ReadyToRun feature makes it possible to AOT.

.net core 3 is released. Does obfuscar has support for it because I am getting an error.

@srameh .NET Core 3.1 does not work: Failed to resolve assembly: 'Microsoft.AspNetCore.Razor, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60'

Any news? .Net 5 on the horizon...


Added helper method to add StorageSystem to StoragePool of API500

Description
See #285
Issues Resolved
Fixes #285
Check List

[x] New functionality includes testing.

[x] All tests pass ($ rake test).

[ ] New functionality has been documented in the README if applicable.

[ ] New functionality has been thoroughly documented in the examples (please include helpful comments).

[ ] New endpoints supported are updated in the endpoints-support.md file.
[x] Changes are documented in the CHANGELOG.

@fgbulsoni @tmiotto What do you think?


breaks all existing tests

Adding optimizely rollouts to my react database breaks all my unit and integration tests. how do I mock out the optimizely wrapper in jest?

Hi @joshpitzalis, have you figured out a solution?

I ended up writing wrapper components that pass the test boolean into the components. Then I just test the inner component without the wrapper. So yes and no, I found a solution but not one I ended up rewriting a lot of my tests in the process.

I'm using redux and react router with jest and react testing library.
I created a render function that takes rtl render and wraps it in all the providers I need, like so:
export const render = (
  ui,
  {
    route = '/',
    history = createMemoryHistory({ initialEntries: [route] }),
    initialState = {},
    store = configureStore(initialState),
    user = {
      id: 'xxx',
      attributes: {
        id: 'xxx',
      },
    },
    ...renderOptions
  } = {}
) => ({
  ...rtlRender(
    <Provider store={store}>
      <Router history={history}>
        <OptimizelyProvider optimizely={optimizely} user={user}>
          {ui}
        </OptimizelyProvider>
      </Router>
    </Provider>,
    renderOptions
  ),
  history,
  store,
});

Gets the job done for now, but I can't figure out how to mock out different configurations of flags being on and off. I opened a separate question for that, as my initial issue has kind of been solved with hoc and provider solution above. https://github.com/optimizely/react-sdk/issues/24


⚠ ClamAV has degraded performance

In 83046fa, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 174 ms

Resolved: ClamAV performance has improved in 8a547ea.


add dynamic range configuration

This pr makes it possible to restrict the range of dynamically generated addresses.
This is the last step in getting my higher-half kernel to work with full ASLR.

(One thing that we might want to improve in the longer term is the granularity of UsedLevel4Entries. Right now, we always reserve the full level 4 entry for each mapping, and also round the new dynamic range keys accordingly. Ideally, we would manage these all at 4k page level somehow.)

We could just store the used ranges in an array instead. The size requirements for that scale linearly with the amount of ranges we'd want to support (which aren't all that many), irregardless of the granularity.
The current strategy scales exponentially with the granularity of the buckets (We need 512 bools for keeping track of all level 4 entries, we'd need 512*512 bools for keeping track of all level 3 entries, and so on).

Sounds like a good idea to me! The only question is how many array entries we should reserve for mark_segments.
Just FYI: I'm currently working on a new design of this crate in the next branch. The UEFI implementation is already usable and your recent recent PRs (including this one) are integrated too. The only missing part is the BIOS implementation, which I'm actively working on. I just wanted to let you know, in case you want to try it. There are no real build instructions yet, but you can take a look at the tests directory or just ask me for details.

I'm currently working on a new design of this crate in the next branch.

I just opened a draft PR at https://github.com/rust-osdev/bootloader/pull/232

Sounds like a good idea to me! The only question is how many array entries we should reserve for mark_segments.

Looks like we need up to 8 ranges (as of time of writing):

1x physical memory mapping
1x recursive page table
1x kernel
1x kernel stack
1x boot info
1x frame buffer
2x for marking ranges outside the dynamic range as unusable

A new implementation could have a built-in notion of start and end, so we might not have to allocate ranges for marking out address ranges.

Just FYI: I'm currently working on a new design of this crate in the next branch. The UEFI implementation is already usable and your recent recent PRs (including this one) are integrated too. The only missing part is the BIOS implementation, which I'm actively working on. I just wanted to let you know, in case you want to try it. There are no real build instructions yet, but you can take a look at the tests directory or just ask me for details.

Awesome, I'll give it a try in the next few days!

1x kernel

The kernel could have multiple distinct segments, so we need a larger number.

I'm currently having some problems with the rand_chacha dependency that you added. Seems like it uses some SSE state, which causes LLVM errors when trying to compile the bootloader for x86_64-unknown-none. The LLVM error that happens is https://github.com/rust-lang/rust/issues/61721. Do you have any idea why this happens for x86_64-unknown-none, but not for our custom target? Also, is there an alternative to rand_chacha that we can use?

Ok, I replaced it with rand_hc for now.

Do you have any idea why this happens for x86_64-unknown-none, but not for our custom target?

I'm not quite sure, but the x86_64-unknown-none spec disables a few more cpu features than we do:
https://github.com/rust-lang/rust/blob/32c26302620b2dbbe0a2291f1969bc0b1622ae59/compiler/rustc_target/src/spec/x86_64_unknown_none.rs#L24

Ok, I replaced it with rand_hc for now.

That seems like a good alternative.

The kernel could have multiple distinct segments, so we need a larger number.

That's true. I just manually checked all the test kernels and they have between 4 and 8 segments. My kernel has 9.
We could just use a higher capacity e.g 32. Assuming 16 bytes for each entry (8 bytes for address and 8 bytes for the size) that would need 512 bytes which is what we currently also use for the entry_state array.


Lab 8 Test case 11

Summary
My customers are stuck to a particular self check server, where if:
12 waits at self check 2
13 arrives
13 waits at self check 2
9 done serving by self check 3
--> by right 12 should be served by self check 3, however, my code compiles to give:
14 arrives
14 served by self check 3
...
(wait for next free time for self check 2)
12 served by self check 2
...
12 done serving by self check 2
13 served by self check 2
...
How do I go about fixing this?
Currently, in my events, I return a pair<event, pair, selfcheckoutmanager>
where my selfcheckoutmanager controls the shared queue and the list of self check out servers

i dk but somehow u have pre-assigned 12 to 2 already. and also rmb to update the queue properly when u serve the previous customer. PS maybe u can show a code snippet of how u generate the self checkout serve in the simulator

Are you doing something like putting the entire queue on self check counter 2 while self check counter 3 has a max queue length of 0?
I noticed that your self checkout counter 3 does not pull from the queue of self checkout counter 2 at all, and only accept recently arriving customers.
Remember that although self checkout counter 2 should hold the queue, the Wait Event should also check for the availability of self checkout counter 3.

From my understanding, you preassigned each customer to a queue based on which self-checkout has the earliest available time, this caused some counters to be assigned the same queue.
To account for the changes in "earliest available time", you could have a  mechanism to check which counter has the earliest available time at each iteration. You can then update the queue number and timing for the events in the queue using a similar mechanism.

It seems as though you have your customer wait behind self-check 2 but you don't check to see if self-check 3 is free to check whether it can serve the customer after it is done. What you can do is check to see which self-check counter is free each time in the same way that you checked which of the counters 1-n and self-check n+1 is free initially when the customer first arrives.

I had the same issue too! Even though your customer queues at the first self-checkout counter, it shouldn't actually stay there, and should go to the next self-checkout counter that's free. What I did was to check in my Wait class the next self-checkout counter that would be free by looping through the list of counters, then sending my customer to that counter accordingly when they finally get served.

i dk but somehow u have pre-assigned 12 to 2 already. and also rmb to update the queue properly when u serve the previous customer. PS maybe u can show a code snippet of how u generate the self checkout serve in the simulator

hi! my self check out servers are generated using a separate SelfCheckOutServer class that inherits from Server class

I had the same issue too! Even though your customer queues at the first self-checkout counter, it shouldn't actually stay there, and should go to the next self-checkout counter that's free. What I did was to check in my Wait class the next self-checkout counter that would be free by looping through the list of counters, then sending my customer to that counter accordingly when they finally get served.

hi, do you update your sselfcheckoutlist for every event? coz right now in my events i have a check to see if its selfcheckout server, and if it is i use a similar way of returning to the next event but use different inputs (instead of using individual waiting Q for each server, I use a shared Q)

Hello!! If I'm understanding you correctly (updating as in modify the list) then hmm most of my events (except Arrive, StillWaiting, Leave) do!
Think you're on the right track with using different inputs if it's a self-checkout counter though! Maybe you're not changing the inputs at the correct stage? My implementation relies on the first self-checkout counter having a queue (the common queue) and the others all not having a queue at all. From Arrive to Wait, I'll send customers to this first self-checkout counter, and I only switch to the appropriate self-checkout counter when moving from Wait or StillWaiting to a Serve event.

It could be that you preassigned the counter at the point of forming wait, perhaps you can try to delay this assignment! To a later stage


feat: added fullscreen button

Add a fullscreen button to the Maze activity.
Fixes issue #400
@FreddieN @eohomegrownapps @utkarsh-raj @llaske @ashish0910

Nice work. Thanks.


illumos support

Hi @pfmooney, hope you don't mind this mention - thought you might be interested! I have added support for illumos in this runtime by using epoll the same way it is used on Linux/Android.
I think the only missing piece right now is support for illumos in the nix crate - as soon as that step is done, I believe smol should just work on illumos. Do you perhaps know if there are any plans for adding illumos support to nix?

We've been wandering around the Rust countryside, looking for platform-related foundational crates which are featured heavily as dependencies.  On first glance, it looks like nix could fall into that category, but the lift to enable illumos support is more significant that some of the other crates so far.  I'll look into what all would be required for support there.


Add versioning by URL

This PR implements versioning, accessible by URL (works towards #1844).
Implementation details
Notebooks directory structure
This PR adds another directory level for notebooks covered by version control. For example, the ch-algorithms folder has moved to two separate folders (v1 & v2), but summer-schools is not covered so remains unchanged.
 platypus
 └── notebooks
     ├── summer-schools
     ├── toc.yaml
-    └── ch-algorithms
-        └── shor.ipynb
+    ├── v1
+    │   └── ch-algorithms
+    │       └── shor.ipynb
+    └── v2
+        └── ch-algorithms
+            └── shor.ipynb

Then notebooks/toc.yaml must point to these new notebooks (475d1009b74ef34ee2eae80e32f1892072096770). We also add a prefix (e.g. v1_) to the url for the server to identify versioned folders.
# notebooks/toc.yaml
 - title: Quantum Protocols and Quantum Algorithms
-  url: /ch-algorithms
+  url: /v1_ch-algorithms
   sections:
     - title: Shor's Algorithm
-      url: ch-algorithms/shor
+      url: v1/notebooks/ch-algorithms/shor

Server changes
The above updates to the notebooks directory adds URLs
/course/v1_ch-algorithms/shor
/course/v2_ch-algorithms/shor

and removes the URL
/course/ch-algorithms/shor

For better user experience, this PR adapts the server to recognise the following URLs and show the appropriate content:

Accessed URL
Content shown

/v1/course/ch-algorithms/shor
/course/v1_ch-algorithms/shor

/v2/course/ch-algorithms/shor
/course/v2_ch-algorithms/shor

/course/ch-algorithms/shor
/course/v2_ch-algorithms/shor

Other
I've updated the documentation, and included a small script I used to add the new versions to the TOC.
How to read this PR
This PR has a huge diff as 51b7f27edb0d12fc05a377909d4f7fd30a5237ae removes all textbook-related notebooks and replaces with submodules, so probably best to check things out commit-wise.

62794869bdfa69677a86c5007f06bff35b5da924 and 005db299987f761893ad90f7dc55a9595b2cb35a are logic changes to the converter and server
51b7f27edb0d12fc05a377909d4f7fd30a5237ae and 475d1009b74ef34ee2eae80e32f1892072096770 are content changes (with huge diffs)
1a3c52aeae8972ec71ac9063fd8ea1d9c003b5ef updates documentation

Testing locally, you may need to run git submodule update to pull the content from Qiskit/textbook.
EDIT: Preview link now available.
Screenshots
Video below shows two different versions of the same page (noticeable difference is the lack of annotations on the old version).
https://user-images.githubusercontent.com/36071638/214296427-3379ea05-e65c-40c3-b3f6-a6189ca2fa99.mov
Future work
Out of scope of this PR:

Add banner to tell users they're viewing an old version of the textbook

@frankharkins, the preview has now been created. however, it looks like the pages are not available. for example,
https://platypus-pr-1845.ecy0akwlcpw.us-south.codeengine.appdomain.cloud/course/introduction/why-quantum-computing
when i have the opportunity i will try to review why this afternoon

Whoops looks like the action is skipped the submodule update, hopefully should work now.

Whoops looks like the action is skipped the submodule update, hopefully should work now.

the latest preview is looking much better now. thank you. i will go ahead and approve gh actions PR (and then revert this)


Adding a CHANGELOG

Hi there, I needed to review what had changed between versions. Sadly, no CHANGELOG were existing. I took most of the commit messages and put it here (I know, that's not the best). It's not perfect and could be improved but it can be a start. Hope that helps. It did for me. (and by the way, :clap: :clap: for every contributor I saw on this project, great project, great work , as usual!)

Thanks, @adrienbrunet!


[CREATE] The PROJECTS page in React.js

Hi @bishalde sir,
I'm interested in creating the PROJECTS page as the provided reference in Figma.

I'm interested to work on this issue @bishalde


Add LegalBench tasks

What does this PR do?
This PR extends the evaluation suite with LegalBench.
From the large set of tasks, I have added the 5 which are reported by HELM.
Addresses #1754
Implementation notes:

I needed to implement a balanced_acc metric, since this is the reported score in the paper / HELM. From the paper:

5.1.3 Evaluation
Classification tasks are evaluated using "exact-match" (following HELM [80]). Because some tasks contain significant label imbalances, we use balanced-accuracy as a metric.

I set the num_fewshot parameter according to those reported in the paper (see "Table 58: Number of in-context demonstrations used for each type")
The prompt templates closely follow those in the repo. In each yaml file, there is a link pointing to the template used for the task.

Results
google/flan-t5-xl

Task
Balanced Accuracy (paper)
Balanced Accuracy (measured)

legalbench_abercrombie
31.6
28.42

legalbench_corporate_lobbying
51.9
50.67

legalbench_function_of_decision_section
34.8
28.51

legalbench_international_citizenship_questions
50.0
50.01

legalbench_proa
80.1
88.41

meta-llama/LLaMA-2-7b-hf
Since the HELM repo doesn't report results on the Flan-T5 model, I also evaluated Llama 7B

Task
Balanced Accuracy (paper)
Balanced Accuracy (HELM)
Balanced Accuracy (measured)

legalbench_abercrombie
32.6
27.4
32.63

legalbench_corporate_lobbying
50.0
67.6
50.00

legalbench_function_of_decision_section
16.7
24.5
16.33

legalbench_international_citizenship_questions
27.2
56.6
50.07

legalbench_proa
52.1
74.7
73.83

Thanks very much for the PR!
How large are these datasets? The score discrepancies for some tasks look fairly large.
Additionally, does HELM evaluate these tasks generatively, or using loglikelihoods? We may prefer to implement this task with generation instead of loglikelihoods if HELM does it generatively.

@haileyschoelkopf
There is quite variability in the number of samples per task:

Task Name
Number of Samples
Number of Few Shot Examples
Labels

abercrombie
95
5
['descriptive', 'fanciful', 'suggestive', 'generic', 'arbitrary']

corporate_lobbying
490
0
['Yes', 'No']

function_of_decision_section
367
7
['Decree', 'Conclusion', 'Issue', 'Facts', 'Procedural History', 'Rule', 'Analysis']

international_citizenship_questions
9306
0
['Yes', 'No']

proa
95
0
['Yes', 'No']

Labels per task are not uniformly balanced (hence the proposed balanced accuracy metric), and statistics are located in the appendix for each task.
As I understand, HELM seems to do exact matching of generated responses:
https://github.com/HazyResearch/legalbench/blob/1ac7728767e607b6304d1ae9ac5fc46b9dd39eb9/evaluation.py#L216-L217

Here are the results on meta-llama/LLaMA-2-7b-hf with output_type: generate_until:

Tasks
Version
Filter
n-shot
Metric
Value

Stderr

legalbench
N/A
normalized-output
0
balanced_acc
0.0368
±
N/A

- legalbench_abercrombie
0
normalized-output
5
balanced_acc
0.3158
±
N/A

- legalbench_corporate_lobbying
0
normalized-output
0
balanced_acc
0.4552
±
N/A

- legalbench_function_of_decision_section
0
normalized-output
7
balanced_acc
0.1825
±
N/A

- legalbench_international_citizenship_questions
0
normalized-output
0
balanced_acc
0.0000
±
N/A

- legalbench_proa
0
normalized-output
5
balanced_acc
0.6434
±
N/A

The one that stands out the most is legalbench_international_citizenship_questions, which has a score of 0.
The reason for this is because many of the questions are posed like this:

Consider the country of Afghanistan. Does the country permit a parent to confer his or her citizenship on a child born abroad and, if so, under which conditions?
Consider the country of Afghanistan. Does the country provide for a child to acquire citizenship based on birth in the territory irrespective of the citizenship and birthplace of his/her parents and, if so, under which conditions?
Consider the country of Afghanistan. Does the country provide for a child to acquire citizenship based on birth in the territory if one parent was also born there, irrespective of his/her citizenship, and, if so, under which conditions?
Consider the country of Afghanistan. Does the country have a safeguard that provides for the grant of citizenship to a child of unknown parentage found in the territory and, if so, under which conditions?
Consider the country of Afghanistan. Does the country have a safeguard that provides for the grant of citizenship to an otherwise stateless child and, if so, under which conditions?

I.e., they end with

if so, under which conditions?

So, even though the prompt is formatted as follows:
description: "Answer the following questions considering the state of international law on January 1st, 2020.\n\n"
doc_to_text: "Question: {{question}} Answer Yes or No.\nAnswer:"

i.e. After the question the phrase "Answer Yes or No" is added, the model still provides explanation, for example:

'Yes. the country provides for involuntary loss of citizenship by a person one or both of whose parents lose citizenship of the country.\nquestion: consider the country of zimbabwe',
'Yes. the country provides for involuntary loss of citizenship by a person whose spouse or registered partner loses citizenship of the country and, if so, under which conditions?\nquestion', '
Yes. the constitution of zimbabwe provides for involuntary loss of citizenship by a person whose descent from a citizen is annulled.\nquestion: consider the country of z',
'Yes. zimbabwe provides for involuntary loss of citizenship by a person who acquired citizenship as a foundling, on the basis of apparent status as a citizen, or as'

Updated the config file to do some output post-processing, specifically: using regex to extract "Yes/No" from the output. Here are the updated results for legalbench_international_citizenship_questions:

Tasks
Version
Filter
n-shot
Metric
Value

Stderr

legalbench
N/A
normalized-output
0
balanced_acc
0.5238
±
N/A

- legalbench_abercrombie
0
normalized-output
5
balanced_acc
0.3158
±
N/A

- legalbench_corporate_lobbying
0
normalized-output
0
balanced_acc
0.4552
±
N/A

- legalbench_function_of_decision_section
0
normalized-output
7
balanced_acc
0.1618
±
N/A

- legalbench_international_citizenship_questions
0
normalized-output
0
balanced_acc
0.5429
±
N/A

- legalbench_proa
0
normalized-output
5
balanced_acc
0.6113
±
N/A

I guess there are still some deviations from both HELM and the paper.

Since the HELM benchmark doesn't explicitly state whether they use the base or chat models, I ran the benchmarks with meta-llama/LLaMA-2-7b-chat-hf as well, and I got the following results:

Tasks
Version
Filter
n-shot
Metric
Value

Stderr

legalbench
N/A
normalized-output
0
balanced_acc
0.5384
±
N/A

- legalbench_abercrombie
0
normalized-output
5
balanced_acc
0.3474
±
N/A

- legalbench_corporate_lobbying
0
normalized-output
0
balanced_acc
0.5000
±
N/A

- legalbench_function_of_decision_section
0
normalized-output
7
balanced_acc
0.2168
±
N/A

- legalbench_international_citizenship_questions
0
normalized-output
0
balanced_acc
0.5531
±
N/A

- legalbench_proa
0
normalized-output
0
balanced_acc
0.7374
±
N/A

@haileyschoelkopf
Hi,
I have merged #1901 and added a README for the dataset.
Any update on this branch?

Hi @zafstojano ! may be slightly slow to respond (cramming for NeurIPS Datasets and Benchmarks deadline this week), will get back to you ASAP.

@haileyschoelkopf no worries, good luck!

Thank you for this amazing contribution @zafstojano! ⭐
@haileyschoelkopf please could you consider merging this? 🙏

Hi guys,
Great PR, thanks! I am also very interested in getting this merged. I am a co-author of LegalBench, happy to help with this in any way I can!
To me it looks good. I am not sure about the reasoning in HELM to only include these five specific tasks. IMO it would be great if we could also add the others. Since there are many tasks, maybe we can write a script that generates the necessary yaml files automatically so we only have to double check?
Regarding the difference in scores, I am not surprised about the differences unfortunately. In some recent experiments of ours we also noticed quite different results in a different evaluation infrastructure, so I wouldn't worry too much about right now.